本文介绍了Spark pyspark与Spark提交的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

有关spark-submit的文档中的内容如下:

The documentation on spark-submit says the following:

关于pyspark,它表示以下内容:

Regarding the pyspark it says the following:

这个问题听起来很愚蠢,但是当我通过pyspark运行命令时,它们也在集群"上运行,对吗?它们不只在主节点上运行,对吧?

This question may sound stupid, but when i am running the commands though pyspark they also run on the "cluster", right? They do not run on the master node only, right?

推荐答案

这两者之间没有实际区别.如果未配置,则两者都将在本地模式下执行代码.如果配置了master(通过--master命令行参数或spark.master配置),则将使用相应的群集执行程序.

There is no practical difference between these two. If not configured otherwise both will execute code in a local mode. If master is configured (either by --master command line parameter or spark.master configuration) corresponding cluster will be used to execute the program.

这篇关于Spark pyspark与Spark提交的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!

08-24 03:36