获取DStream的第一个元素(take函数)

问题描述

我正在寻找一种方法来检索创建为以下内容的DStream的第一个元素:

I look for a way to retrieve the first elements of a DStream created as:

val dstream = ssc.textFileStream(args(1)).map(x => x.split(",").map(_.toDouble))

不幸的是，dstream上没有 take 功能(如RDD上)//dstream.take(2)!!!

Unfortunately, there is no take function (as on RDD) on a dstream //dstream.take(2) !!!

有人对如何做有任何想法吗?谢谢

Could someone has any idea on how to do it ?! thanks

推荐答案

由于 take()方法返回一个不可序列化的Array，因此先前建议的解决方案对我而言没有编译，因此Spark流将失败，并显示 java.io.NotSerializableException .

The previous suggested solution did not compile for me as the take() method returns an Array, which is not serializable thus Spark streaming will fail with a java.io.NotSerializableException.

对我有用的先前代码的一个简单变体:

A simple variation on the previous code that worked for me:

val n = 10
val partOfResult = dstream.transform(rdd => {
  rdd.filter(rdd.take(n).toList.contains)
})
partOfResult.print

这篇关于获取DStream的第一个元素(take函数)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！