且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

如何迭代scala wrapperArray? (火花)

更新时间:2023-10-20 16:41:16

这是解决此问题的一种方法.

Here is one way to solve this.

import org.apache.spark.sql.Row
import org.apache.spark.sql.functions._
import scala.collection.mutable.WrappedArray

val data = Seq((Seq(1,2,3),Seq(4,5,6),Seq(7,8,9)))
val df = sqlContext.createDataFrame(data)
val first = df.first

// use a pattern match to deferral the type
val mapped = first.getAs[WrappedArray[Int]](0)

// now we can use it like normal collection
mapped.mkString("\n")

// get rows where has array
val rows = df.collect.map {
    case Row(a: Seq[Any], b: Seq[Any], c: Seq[Any]) => 
        (a, b, c)
}
rows.mkString("\n")