且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

火花提交选项列表

更新时间:2023-10-20 17:21:04

虽然 @ suj1th 的宝贵输入确实解决了我的问题,但我正在回答自己的问题以直接解决我的查询./p>


  • 您无需为给定的Spark 属性(配置设置)查找SparkSubmitOptionParser属性名称.两者都很好.但是,请注意,两者的用法之间有细微的差别,如下所示:

    spark-submit --executor-cores 2

    spark-submit --conf spark.executor.cores=2

    上面显示的两个命令将具有相同的效果.第二种方法采用格式--conf <key>=<value>配置.

  • 用引号引起来的值(如果不正确/不完整,请纠正我)

    (i)值不必用任何形式的引号(单引号''或双引号"")括起来(如果需要,也可以).

    (ii)如果值具有space字符,则将整个内容用双引号""括起来,如"<key>=<value>"所示,如 Spark配置

  • YARN 上运行Spark

There are a ton of tunable settings mentioned on Spark configurations page. However as told here, the SparkSubmitOptionParser attribute-name for a Spark property can be different from that property's-name.

For instance, spark.executor.cores is passed as --executor-cores in spark-submit.


Where can I find an exhaustive list of all tuning parameters of Spark (along-with their SparkSubmitOptionParser property name) that can be passed with spark-submit command?

While @suj1th's valuable inputs did solve my problem, I'm answering my own question to directly address my query.


  • You need not look up for SparkSubmitOptionParser's attribute-name for a given Spark property (configuration setting). Both will do just fine. However, do note that there's a subtle difference between there usage as shown below:

    spark-submit --executor-cores 2

    spark-submit --conf spark.executor.cores=2

    Both commands shown above will have same effect. The second method takes configurations in the format --conf <key>=<value>.

  • Enclosing values in quotes (correct me if this is incorrect / incomplete)

    (i) Values need not be enclosed in quotes (single '' or double "") of any kind (you still can if you want).

    (ii) If the value has a space character, enclose the entire thing in double quotes "" like "<key>=<value>" as shown here.

  • For a comprehensive list of all configurations that can be passed with spark-submit, just run spark-submit --help

  • In this link provided by @suj1th, they say that:

    configuration values explicitly set on a SparkConf take the highest precedence, then flags passed to spark-submit, then values in the defaults file.

    If you are ever unclear where configuration options are coming from, you can print out fine-grained debugging information by running spark-submit with the --verbose option.


Following two links from Spark docs list a lot of configurations: