且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

写入 hdfs 路径时出现错误 java.io.IOException: Failed to rename

更新时间:2023-01-13 07:38:59

您可以在一个作业中完成所有选择,在一个表中获取所有选择和联合.

You can do all the selects in one single job, get all the selects and union in a single table.

Dataset<Row> resultDs = givenItemList.parallelStream().map( item -> {   
    String query = "select $item  as itemCol , avg($item) as mean groupBy year";
    return sparkSession.sql(query);
}).reduce((a, b) -> a.union(b)).get

saveDsToHdfs(hdfsPath, resultDs );