我可以在Scalding map方法中输出集合而不是元组吗?

问题描述

如果要在Scalding中使用较小的字段创建一个包含22个以上字段的管道，则受Scala元组的限制，该字段不能包含22个以上的项目.

If you want to create a pipe with more than 22 fields from a smaller one in Scalding you are limited by Scala tuples, which cannot have more than 22 items.

有没有一种使用集合而不是元组的方法?我想象下面的示例中的内容类似，但不幸的是它不起作用:

Is there a way to use collections instead of tuples? I imagine something like in the following example, which sadly doesn't work:

input.read.mapTo('line -> aLotOfFields) { line: String =>
  (1 to 24).map(_.toString)
}.write(output)

推荐答案

实际上可以.在FAQ中- https://github.com/twitter/scalding/wiki/Frequently-asked-questions#what-if-i-have-than-than-22-fields-in-my-data -设置

actually you can. It's in FAQ - https://github.com/twitter/scalding/wiki/Frequently-asked-questions#what-if-i-have-more-than-22-fields-in-my-data-set

val toFields = (1 to 24).map(f => Symbol("field_" + f)).toList

input
  .read
  .mapTo('line -> toFields) { line: String =>
    new Tuple((1 to 24).map(_.toString).map(_.asInstanceOf[AnyRef]): _*)

  }

最后一张地图(_.asInstanceOf [AnyRef])看起来很丑，所以如果您找到更好的解决方案，请告诉我.

the last map(_.asInstanceOf[AnyRef]) looks ugly so if you find better solution let me know please.

这篇关于我可以在Scalding map方法中输出集合而不是元组吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！