Spark Streaming是否支持共享变量?

问题描述

Spark为两种常用用法模式提供了两种有限类型的共享变量:广播变量和累加器，Spark Streaming是否支持它们?

Spark provide two limited types of shared variables for two common usage patterns: broadcast variables and accumulators, are they supported in Spark Streaming??

推荐答案

是的，您可以像通常通过 SparkContext 那样使用它们.唯一的区别是您需要从 SparkStreamingContext 中获取它:

Yes, you can use them as you'd normally do via SparkContext. The only difference is that you need to get it from SparkStreamingContext:

val sparkConf = new SparkConf().setAppName("MyApp")
val ssc = new StreamingContext(sparkConf, Seconds(1))
ssc.sparkContext.broadcast(myValue)

使用Spark Streaming时，您可能需要在某个时候更新该值，为此，您应该使用 unpersist()(或阻止版本)，然后再次播放.

With spark streaming you might want to update that value at some point, to do that you should use unpersist() (or the blocking version) and broadcast again.

这篇关于Spark Streaming是否支持共享变量?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！