针对RDD, 使用 keyBy 来构筑 key-line 对：

[training@localhost ~]$ cat webs.log

56.31.230.188 - 90700 "GET/KDDOC-00101.html HTTP/1.0"
56.32.230.186 - 90700 "GET/contents.css HTTP/1.0"
202.156.27.99 - 25223 "GET /KDDOC-00220.html HTTP/1.0"
[training@localhost ~]$ 
[training@localhost ~]$ hdfs dfs -put webs.log
[training@localhost ~]$ 
[training@localhost ~]$ hdfs dfs -cat webs.log
56.31.230.188 - 90700 "GET/KDDOC-00101.html HTTP/1.0"
56.32.230.186 - 90700 "GET/contents.css HTTP/1.0"
202.156.27.99 - 25223 "GET /KDDOC-00220.html HTTP/1.0"
[training@localhost ~]$ 
[training@localhost ~]$

In [23]: mylogs = sc.textFile("webs.log")

In [25]: mylogs001 = mylogs.keyBy(lambda line: line.split(' ')[2])

In [26]: mylogs001.take(1)
Out[26]: [(u'90700', u'56.31.230.188 - 90700 "GET/KDDOC-00101.html HTTP/1.0"')]

In [28]: mylogs001.take(2)
Out[28]: 
[(u'90700', u'56.31.230.188 - 90700 "GET/KDDOC-00101.html HTTP/1.0"'),
(u'90700', u'56.32.230.186 - 90700 "GET/contents.css HTTP/1.0"')]

作一个对比，看看 mylogs001.take(3) 和 mylogs.take(3)

In [30]: mylogs001.take(3)
Out[30]: 
[(u'90700', u'56.31.230.188 - 90700 "GET/KDDOC-00101.html HTTP/1.0"'),
(u'90700', u'56.32.230.186 - 90700 "GET/contents.css HTTP/1.0"'),
(u'25223', u'202.156.27.99 - 25223 "GET /KDDOC-00220.html HTTP/1.0"')]

In [31]: mylogs.take(3)
Out[31]: 
[u'56.31.230.188 - 90700 "GET/KDDOC-00101.html HTTP/1.0"',
u'56.32.230.186 - 90700 "GET/contents.css HTTP/1.0"',
u'202.156.27.99 - 25223 "GET /KDDOC-00220.html HTTP/1.0"']

本文转自健哥的数据花园博客园博客，原文链接：http://www.cnblogs.com/gaojian/p/008-Aggregating-Data-with-Pair-RDDs-keyBy.html，如需转载请自行联系原作者