您可以通过使用 from_json 函数将您的 json 字符串转换为实际的 json 而受益.为此，您必须定义一个与您的 json 字符串匹配的 schema .最后使用 explode 函数将 struct 数组 分隔到不同的行，就像使用 eval 一样.

You can benefit by using from_json function to convert your json string to actual json. For that you will have to define a schema matching to your json string. And finally use explode function to separate the struct array to different rows as you did with eval.

如果你有一个数据

x = "[{u'date': u'2015-02-08', u'by': u'abc@gg.com', u'value': u'NA'}, {u'date': u'2016-02-08', u'by': u'dfg@yaa.com', u'value': u'applicable'}, {u'date': u'2017-02-08', u'by': u'wrwe@hot.com', u'value': u'ufc'}]"

然后创建dataframe

df = sqlContext.createDataFrame([(x,),], ["x"])

+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|x                                                                                                                                                                                                              |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|[{u'date': u'2015-02-08', u'by': u'abc@gg.com', u'value': u'NA'}, {u'date': u'2016-02-08', u'by': u'dfg@yaa.com', u'value': u'applicable'}, {u'date': u'2017-02-08', u'by': u'wrwe@hot.com', u'value': u'ufc'}]|
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


root
 |-- x: string (nullable = true)

使用 jsons

正如我所解释的，你需要一个 schema、regexp_replace 函数、from_json 函数和 explode 函数作为

As I had explained, you would need a schema, regexp_replace function, from_json function and explode function as

from pyspark.sql import types as T
schema = T.ArrayType(T.StructType([T.StructField('date', T.StringType()), T.StructField('by', T.StringType()), T.StructField('value', T.StringType())]))

from pyspark.sql import functions as F
df = df.withColumn("x", F.explode(F.from_json(F.regexp_replace(df['x'], "(u')", "'"), schema=schema)))

应该给你

+-----------------------------------+
|x                                  |
+-----------------------------------+
|[2015-02-08,abc@gg.com,NA]         |
|[2016-02-08,dfg@yaa.com,applicable]|
|[2017-02-08,wrwe@hot.com,ufc]      |
+-----------------------------------+

root
 |-- x: struct (nullable = true)
 |    |-- date: string (nullable = true)
 |    |-- by: string (nullable = true)
 |    |-- value: string (nullable = true)

如果您需要问题中提到的 json 字符串，那么您可以使用 to_json 函数作为

If you require the json strings as mentioned in the question then you can use to_json function as

df = df.withColumn("x", F.to_json(df['x']))

这会给你

+-------------------------------------------------------------+
|x                                                            |
+-------------------------------------------------------------+
|{"date":"2015-02-08","by":"abc@gg.com","value":"NA"}         |
|{"date":"2016-02-08","by":"dfg@yaa.com","value":"applicable"}|
|{"date":"2017-02-08","by":"wrwe@hot.com","value":"ufc"}      |
+-------------------------------------------------------------+

仅使用字符串

如果您不想经历 json 的所有复杂性，那么您可以简单地使用字符串.为此，您需要 嵌套 regex_replace、split 和 explode 函数作为

If you don't want to go through all the complexities of jsons then you can simply work with strings. For that you would need nested regex_replace, split and explode functions as

from pyspark.sql import functions as F
df = df.withColumn("x", F.explode(F.split(F.regexp_replace(F.regexp_replace(F.regexp_replace(df['x'], "(u')", "'"), "[\\[\\]\s]", ""), "},\\{", "};&;{"), ";&;")))

应该给你

+-------------------------------------------------------------+
|x                                                            |
+-------------------------------------------------------------+
|{'date':'2015-02-08','by':'abc@gg.com','value':'NA'}         |
|{'date':'2016-02-08','by':'dfg@yaa.com','value':'applicable'}|
|{'date':'2017-02-08','by':'wrwe@hot.com','value':'ufc'}      |
+-------------------------------------------------------------+