问题描述
DStream
可以具有 type参数
s吗?
如果是,怎么办?
当我在 myDStream上尝试
(类参数),我得到: lazy val qwe = mStream.mapWithState(stateSpec)
时:DStream [(A,B)]
When I try lazy val qwe = mStream.mapWithState(stateSpec)
on myDStream: DStream[(A, B)]
(class parameter), I get:
value mapWithState is not a member of org.apache.spark.streaming.dstream.DStream[(A, B)]
lazy val qwe = mStream.mapWithState(stateSpec)
推荐答案
Spark API的大量子集需要隐式的 ClassTags
(请参阅)和 PairDStreamFunctions.mapWithState
一样.检查类定义:
Substantial subset of the Spark API requires implicit ClassTags
(see Scala: What is a TypeTag and how do I use it?) and PairDStreamFunctions.mapWithState
is no different. Check class definition:
class PairDStreamFunctions[K, V](self: DStream[(K, V)])
(implicit kt: ClassTag[K], vt: ClassTag[V], ord: Ordering[K])
和:
def mapWithState[StateType: ClassTag, MappedType: ClassTag](
spec: StateSpec[K, V, StateType, MappedType]
): MapWithStateDStream[K, V, StateType, MappedType] = {
...
}
如果要创建对通用对流进行操作并使用 mapWithState
的函数,则应至少为 KeyType
和 ValueType
类型:
If want to create a function which operates on a generic pair streams and uses mapWithState
you should at least provide ClassTags
for KeyType
and ValueType
types:
def foo[T : ClassTag, U : ClassTag](
stream: DStream[(T, U)], f: StateSpec[T, U, Int, Int]) = stream.mapWithState(f)
如果同时设置了 StateType
和 MappedType
的参数,则您还需要 ClassTags
:
If StateType
and MappedType
are parametrized as well you'll need ClassTags
for these too:
def bar[T : ClassTag, U : ClassTag, V : ClassTag, W : ClassTag](
stream: DStream[(T, U)], f: StateSpec[T, U, V, W]) = stream.mapWithState(f)
这篇关于对DStream进行类型参数化的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!