我有rdd项目,例如:

(3922774869,10,1)
(3922774869,11,1)
(3922774869,12,2)
(3922774869,13,2)
(1779744180,10,1)
(1779744180,11,1)
(3922774869,14,3)
(3922774869,15,2)
(1779744180,16,1)
(3922774869,12,1)
(3922774869,13,1)
(1779744180,14,1)
(1779744180,15,1)
(1779744180,16,1)
(3922774869,14,2)
(3922774869,15,1)
(1779744180,16,1)
(1779744180,17,1)
(3922774869,16,4)
...

代表(id, age, count),我想对这些行进行分组以生成一个数据集,为此,每一行都代表每个id的年龄分布,如下所示((id, age)是uniq):
(1779744180, (10,1), (11,1), (12,2), (13,2) ...)
(3922774869, (10,1), (11,1), (12,3), (13,4) ...)

这是(id, (age, count), (age, count) ...)
有人可以给我一个提示吗?

最佳答案

您可以先按两个字段进行归约,然后使用groupBy:

rdd
  .map { case (id, age, count) => ((id, age), count) }.reduceByKey(_ + _)
  .map { case ((id, age), count) => (id, (age, count)) }.groupByKey()

对于上面的输入,它将返回RDD[(Long, Iterable[(Int, Int)])],其中将包含以下两个记录:
(1779744180,CompactBuffer((16,3), (15,1), (14,1), (11,1), (10,1), (17,1)))
(3922774869,CompactBuffer((11,1), (12,3), (16,4), (13,3), (15,3), (10,1), (14,5)))

10-04 10:32