且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

Spark将列合并为嵌套数组

更新时间:2023-11-18 19:03:04

如果要将多个列合并到ArrayType的新列中,可以使用 array 函数:

If you want to combine multiple columns into a new column of ArrayType, you can use the array function:

import org.apache.spark.sql.functions._
val result = inputSmall.withColumn("combined", array($"transformedCol1", $"transformedCol2"))
result.show()

+-------+---------------+-------+---------------+-----------+
|column1|transformedCol1|column2|transformedCol2|   combined|
+-------+---------------+-------+---------------+-----------+
|      A|            0.3|      B|           0.25|[0.3, 0.25]|
|      A|            0.3|      g|            0.4| [0.3, 0.4]|
|      d|            0.0|      f|            0.1| [0.0, 0.1]|
|      d|            0.0|      d|            0.7| [0.0, 0.7]|
|      A|            0.3|      d|            0.7| [0.3, 0.7]|
|      d|            0.0|      g|            0.4| [0.0, 0.4]|
|      c|            0.2|      B|           0.25|[0.2, 0.25]|
+-------+---------------+-------+---------------+-----------+