更新时间:2023-10-20 17:21:04
虽然 @ suj1th 的宝贵输入确实解决了我的问题,但我正在回答自己的问题以直接解决我的查询./p>
您无需为给定的Spark
属性(配置设置)查找SparkSubmitOptionParser
的属性名称.两者都很好.但是,请注意,两者的用法之间有细微的差别,如下所示:
spark-submit --executor-cores 2
spark-submit --conf spark.executor.cores=2
上面显示的两个命令将具有相同的效果.第二种方法采用格式--conf <key>=<value>
的配置.
用引号引起来的值(如果不正确/不完整,请纠正我)
(i)值不必用任何形式的引号(单引号''
或双引号""
)括起来(如果需要,也可以).
(ii)如果值具有space
字符,则将整个内容用双引号""
括起来,如"<key>=<value>"
所示,如 Spark
配置
YARN
上运行Spark
There are a ton of tunable settings mentioned on Spark
configurations page. However as told here, the SparkSubmitOptionParser
attribute-name for a Spark
property can be different from that property's-name.
For instance, spark.executor.cores
is passed as --executor-cores
in spark-submit
.
Where can I find an exhaustive list of all tuning parameters of Spark
(along-with their SparkSubmitOptionParser
property name) that can be passed with spark-submit
command?
While @suj1th's valuable inputs did solve my problem, I'm answering my own question to directly address my query.
You need not look up for SparkSubmitOptionParser
's attribute-name for a given Spark
property (configuration setting). Both will do just fine. However, do note that there's a subtle difference between there usage as shown below:
spark-submit --executor-cores 2
spark-submit --conf spark.executor.cores=2
Both commands shown above will have same effect. The second method takes configurations in the format --conf <key>=<value>
.
Enclosing values in quotes (correct me if this is incorrect / incomplete)
(i) Values need not be enclosed in quotes (single ''
or double ""
) of any kind (you still can if you want).
(ii) If the value has a space
character, enclose the entire thing in double quotes ""
like "<key>=<value>"
as shown here.
For a comprehensive list of all configurations that can be passed with spark-submit
, just run spark-submit --help
In this link provided by @suj1th, they say that:
configuration values explicitly set on a SparkConf take the highest precedence, then flags passed to spark-submit, then values in the defaults file.
If you are ever unclear where configuration options are coming from, you can print out fine-grained debugging information by running spark-submit with the --verbose option.
Following two links from Spark
docs list a lot of configurations: