且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

在Spark中将DataFrame转换为Json数组

更新时间:2022-05-10 04:31:20

data1.schema.json将为您提供一个JSON字符串,其中包含数据框的架构,而不是实际数据本身.您会得到:

data1.schema.json will give you a JSON string containing the schema of the dataframe and not the actual data itself. You will get :

String = {"type":"struct",
          "fields":
                  [{"name":"A","type":"integer","nullable":false,"metadata":{}},
                  {"name":"B","type":"string","nullable":true,"metadata":{}}]}

要将数据帧转换为JSON数组,您需要使用DataFrame的toJSON方法:

To convert your dataframe to array of JSON, you need to use toJSON method of DataFrame:

val df = sc.parallelize(Array( (1, "test"), (2, "mytest") )).toDF("A", "B")
df.show()

+---+------+
|  A|     B|
+---+------+
|  1|  test|
|  2|mytest|
+---+------+

df.toJSON.collect.mkString("[", "," , "]" )
String = [{"A":1,"B":"test"},{"A":2,"B":"mytest"}]