且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

Spark中的爆炸结构

更新时间:2023-11-18 20:11:28

这很简单:

val newDF = df.select("uploadedDate", "data.*");

您告诉您选择上载日期,然后选择字段数据的所有子元素

You tell to select uploadedDate and then all subelements of field data

示例:

scala> case class A(a: Int, b: Double)
scala> val df = Seq((A(1, 1.0), "1"), (A(2, 2.0), "2")).toDF("data", "uploadedDate")
scala> val newDF = df.select("uploadedDate", "data.*")
scala> newDF.show()
+------------+---+---+
|uploadedDate|  a|  b|
+------------+---+---+
|           1|  1|1.0|
|           2|  2|2.0|
+------------+---+---+

scala> newDF.printSchema()
root
 |-- uploadedDate: string (nullable = true)
 |-- a: integer (nullable = true)
 |-- b: double (nullable = true)