且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

如何使用spark-submit的--properties-file选项在IntelliJ IDEA中启动Spark应用程序?

更新时间:2021-12-15 17:51:48

认为可以使用--properties-file从IntelliJ IDEA内部启动Spark应用程序.

I don't think it's possible to use --properties-file to launch a Spark application from within IntelliJ IDEA.

spark-submit是用于提交Spark应用程序以执行的Shell脚本,在为Spark应用程序创建适当的提交环境之前,它不需要做任何额外的事情.

spark-submit is the shell script to submit Spark application for execution and does few extra things before it creates a proper submission environment for the Spark application.

不过,您可以利用Spark应用程序默认加载的conf/spark-defaults.conf来模仿--properties-file的行为.

You can however mimic the behaviour of --properties-file by leveraging conf/spark-defaults.conf that a Spark application loads by default.

您可以在src/test/resources(或src/main/resources)下使用内容properties.conf创建一个conf/spark-defaults.conf.那是应该起作用的.

You could create a conf/spark-defaults.conf under src/test/resources (or src/main/resources) with the content of properties.conf. That is supposed to work.