问题描述
我有一个用Apache Beam(使用Spark Runner)开发的流传输管道，该管道从kinesis流中读取数据.
I have a streaming pipeline developed in Apache Beam (using Spark Runner) which reads from kinesis stream.
我正在寻找Apache Beam中用于管理运动学检查点的选项(即定期存储运动学流的当前位置)，以便它使系统从故障中恢复并继续处理流停止的地方.
I am looking out for options in Apache Beam to manage kinesis checkpointing (i.e. stores periodically the current position of kinesis stream) so as it allows the system to recover from failures and continue processing where the stream left off.
是否有可供Apache Beam支持的运动学检查点的设置，类似于Spark Streaming(参考链接- https://spark.apache.org/docs/2.2.0/streaming-kinesis-integration.html )?
Is there a provision available for Apache Beam to support kinesis checkpointing as similar to Spark Streaming (Reference link - https://spark.apache.org/docs/2.2.0/streaming-kinesis-integration.html)?
推荐答案
由于 KinesisIO 基于 UnboundedSource.CheckpointMark ，它使用了Beam提供的标准检查点机制 UnboundedSource.UnboundedReader .
Since KinesisIO is based on UnboundedSource.CheckpointMark, it uses the standard checkpoint mechanism, provided by Beam UnboundedSource.UnboundedReader.
一旦读取了 KinesisRecord (实际上是从实际上通过从Kinesis分片中获取记录的记录队列中提取的记录队列中提取)，则分片检查点将为 SequenceNumber 进行nofollow noreferrer>更新，然后保存，具体取决于 UnboundedSource 的运行程序实现和检查点处理.
Once a KinesisRecord has been read (actually, pulled from a records queue that is feed separately by actually fetching the records from Kinesis shard), then the shard checkpoint will be updated by using the record SequenceNumber and then, depending on runner implementation of UnboundedSource and checkpoints processing, will be saved.
Afaik，Beam Spark Runner 为此目的使用了火花状态机制.
Afaik, Beam Spark Runner uses Spark States mechanism for this purposes.
这篇关于Apache Beam如何管理运动学检查点?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！