且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

使用spark datastax连接器忽略带有数据框的Null

更新时间:2023-11-18 17:35:34

找出解决方案:

文档中存在提示: https://github.com/datastax/spark-cassandra- 设置数据集的连接器特定选项
主题下的connector / blob / master / doc / 14_data_frames.md

A hint is present in the document: https://github.com/datastax/spark-cassandra-connector/blob/master/doc/14_data_frames.md under Setting Connector Specific Options on Datasets topic.

确切的代码如下:

transformedData.write.format("org.apache.spark.sql.cassandra").option("header","false").option("spark.cassandra.output.ignoreNulls", true).mode("append").options(Map( "table" -> table_name, "keyspace" -> keyspace_name)).save()