问题描述
在阅读官方文档之后,我尝试使用 getOrCreate > checkpoint .一些摘要:
After read official docs, i tried using checkpoint
with getOrCreate
in spark streaming. Some snippets:
def get_ssc():
sc = SparkContext("yarn-client")
ssc = StreamingContext(sc, 10) # calc every 10s
ks = KafkaUtils.createDirectStream(
ssc, ['lucky-track'], {"metadata.broker.list": KAFKA_BROKER})
process_data(ks)
ssc.checkpoint(CHECKPOINT_DIR)
return ssc
if __name__ == '__main__':
ssc = StreamingContext.getOrCreate(CHECKPOINT_DIR, get_ssc)
ssc.start()
ssc.awaitTermination()
该代码可以很好地进行恢复,但是已恢复的上下文始终可以在旧的流程函数上运行.这意味着,即使我更改了map/reduce函数代码,也根本无法正常工作.
The code works fine for recover, but the recovered context always works on the old process function. It means that even if i changed map/reduce function code, it not works at all.
直到现在,spark(1.5.2)仍不支持python的任意偏移量.那么,我应该怎么做才能使其正常工作?
Until now, spark(1.5.2) still not support arbitrary offset for python. So, what should i do to make this work properly?
推荐答案
这种行为是设计使然"的,并且对于java/scala Spark应用程序也有效.在检查点时将整个代码序列化.如果代码更改,则检查点数据应被截断.
Such behaviour is "by design", and is valid also for java/scala Spark applications. Entire code is serialized while checkpointing. If code changes, checkpoint data should be truncated.
这篇关于使用Python Spark Direct方法时如何从检查点恢复?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!