collect_list仅在1.6中出现.

我将研究基础的RDD.方法如下:

I'd go through the underlying RDD. Here's how:

data_df.show()
+--------+---+------+----+
|username|qid|row_no|text|
+--------+---+------+----+
|       d|  2|     2|ball|
|       a|  1|     1|this|
|       a|  1|     3|text|
|       a|  1|     2|  is|
|       d|  2|     1| the|
+--------+---+------+----+

然后这个

reduced = data_df\
    .rdd\
    .map(lambda row: ((row[0], row[1]), [(row[2], row[3])]))\
    .reduceByKey(lambda x,y: x+y)\
    .map(lambda row: (row[0], sorted(row[1], key=lambda text: text[0]))) \
    .map(lambda row: (
            row[0][0], 
            row[0][1], 
            ','.join([str(e[0]) for e in row[1]]),
            ' '.join([str(e[1]) for e in row[1]])
        )
    )

schema_red = typ.StructType([
        typ.StructField('username', typ.StringType(), False),
        typ.StructField('qid', typ.IntegerType(), False),
        typ.StructField('row_no', typ.StringType(), False),
        typ.StructField('text', typ.StringType(), False)
    ])

df_red = sqlContext.createDataFrame(reduced, schema_red)
df_red.show()

上面产生了以下内容:

+--------+---+------+------------+
|username|qid|row_no|        text|
+--------+---+------+------------+
|       d|  2|   1,2|    the ball|
|       a|  1| 1,2,3|this is text|
+--------+---+------+------------+

在大熊猫中

df4 = pd.DataFrame([
        ['a', 1, 1, 'this'],
        ['a', 1, 2, 'is'],
        ['d', 2, 1, 'the'],
        ['a', 1, 3, 'text'],
        ['d', 2, 2, 'ball']        
    ], columns=['username', 'qid', 'row_no', 'text'])

df_groupped=df4.sort_values(by=['qid', 'row_no']).groupby(['username', 'qid'])

df3 = pd.DataFrame()
df3['row_no'] = df_groupped.apply(lambda row: ','.join([str(e) for e in row['row_no']]))
df3['text']   = df_groupped.apply(lambda row: ' '.join(row['text']))

df3 = df3.reset_index()