且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

在EMR Spark应用程序中加载道具文件

更新时间:2021-10-23 18:48:04

您不能直接从S3加载属性文件.相反,您需要将属性文件下载到主节点的某个位置,然后提交引用该节点上的本地路径的spark作业.您可以使用command-runner.jar来运行aws cli实用程序进行下载.

You can't load a properties file directly from S3. Instead you will need to download the properties file to your master node somewhere, then submit the spark job referencing the local path on that node. You can do the download by using command-runner.jar to run the aws cli utility.