更新时间:2023-01-13 07:38:59
您可以在一个作业中完成所有选择,在一个表中获取所有选择和联合.
You can do all the selects in one single job, get all the selects and union in a single table.
Dataset<Row> resultDs = givenItemList.parallelStream().map( item -> {
String query = "select $item as itemCol , avg($item) as mean groupBy year";
return sparkSession.sql(query);
}).reduce((a, b) -> a.union(b)).get
saveDsToHdfs(hdfsPath, resultDs );