更新时间:2023-01-23 10:42:25
使用 Databricks Spark-CSV 即可直接保存到CSV文件,然后像这样从CSV文件加载
Using Databricks Spark-CSV you can save directly to a CSV file and load from a CSV file afterwards like this
import org.apache.spark.sql.SQLContext
SQLContext sqlContext = new SQLContext(sc);
DataFrame df = sqlContext.read()
.format("com.databricks.spark.csv")
.option("inferSchema", "true")
.option("header", "true")
.load("cars.csv");
df.select("year", "model").write()
.format("com.databricks.spark.csv")
.option("header", "true")
.option("codec", "org.apache.hadoop.io.compress.GzipCodec")
.save("newcars.csv");