且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

如何在Spark 2.0中使用用户定义类型?

更新时间:2022-12-06 11:36:24

现在您是对的,Spark 2.x不再像Spark 1.x一样具有用作API的任何类型的UDT.

Well you are right for now, the Spark 2.x has no more any kind of UDT to use as an API that was like in Spark 1.x.

您可以在此故障单中看到他们制作的 SPARK-14155 私有创建一个新的API.我们希望自Spark 1.5起开放一个故障单,希望在Spark 2.2中将其关闭. SPARK -7768 .

You can see here in this ticket SPARK-14155 that they make it privet to create a new API. That we have a ticket open since Spark 1.5 that we wish that will be closed in Spark 2.2 SPARK-7768.

好吧,类型现在不足以创建您的UDT,但是...您可以使用一些技巧来将自定义对象设置为DataSet. 此处是一个示例.

Well, types are not good for now to create your UDT but... There few tricks that you can set your custom objects to a DataSet. Here is one example.