且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

为 spark 数据框中的特定列应用逻辑

更新时间:2023-11-18 23:26:16

由于您尝试将 XML 数据列拉出到单独的 DataFrame,您仍然可以使用 spark-xml 包中的代码.你只需要直接使用他们的阅读器.

Since you are trying to pull the XML data column out to a separate DataFrame you can still use the code from spark-xml's package. You just need to use their reader directly.

case class Data(id: Int, accountid: Int, xmldata: String)
val df = Seq(
    Data(1001, 12345, "<AccountSetup xmlns:xsi=\"test\"><Customers test=\"a\">d</Customers></AccountSetup>"),
    Data(1002, 12345, "<AccountSetup xmlns:xsi=\"test\"><Customers test=\"b\">e</Customers></AccountSetup>"),
    Data(1003, 12345, "<AccountSetup xmlns:xsi=\"test\"><Customers test=\"c\">f</Customers></AccountSetup>")
).toDF


import com.databricks.spark.xml.XmlReader

val reader = new XmlReader()

// Set options using methods
reader.withRowTag("AccountSetup")

val rdd = df.select("xmldata").map(r => r.getString(0)).rdd
val xmlDF = reader.xmlRdd(spark.sqlContext, rdd)

但是,从长远来看,philantrovert 建议使用自定义 XML 解析的 UDF 可能会更清晰.阅读器类的参考链接 这里

However, a UDF as philantrovert suggests with custom XML parsing would probably be cleaner in the long run. Reference link for the reader class here