问题描述
遵循此处提到的解决方案 kafka-mirror-maker-失败复制消费者偏移主题.我能够在 DC1(实时 Kafka 集群)和 DC2(备份 Kafka 集群)集群中启动镜像制造商而没有任何错误.
Following to the solution mentioned here kafka-mirror-maker-failing-to-replicate-consumer-offset-topic. I was able to start mirror maker without any error across DC1(Live Kafka cluster) and DC2(Backup Kafka cluster) clusters.
看起来它也可以跨 DC2 集群从 DC1 集群同步 __consumer_offsets 主题.
Looks like it is also able to sync __consumer_offsets topic across DC2 cluster form DC1 cluster.
问题
如果我关闭 DC1 的消费者并将相同的消费者(相同的 group_id)指向 DC2，即使镜像制造商能够同步此主题和分区的偏移量，它也会再次读取相同的消息.
If I close down consumer for DC1 and point same consumer(same group_id) to DC2 it reads the same messages again even though mirror maker is able sync offsets for this topic and partitions.
我可以看到 LOG-END-OFFSET 显示正确，但 CURRENT-OFFSET 仍然指向旧的导致 LAG.
I can see that LOG-END-OFFSET is showing correctly but CURRENT-OFFSET is still pointing to old causing LAG.
示例
- Mirror Maker 仍在 DC2 中运行.
消费者在 DC1 中关闭之前
- Mirror Maker is still running in DC2.
Before consumer shut down in DC1
//DC1 __consumer_offsets topic
+-----------------------------------------------------------------+
| TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG |
+-----------------------------------------------------------------+
| gs.suraj.test.1 0 10626 10626 0 |
| gs.suraj.test.1 2 10619 10619 0 |
| gs.suraj.test.1 1 10598 10598 0 |
+-----------------------------------------------------------------+
停止 DC1 中的消费者
Stop consumer in DC1
消费者在 DC2 中启动之前
Before consumer start up in DC2
//DC2 __consumer_offsets topic
+-----------------------------------------------------------------+
| TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG |
+-----------------------------------------------------------------+
| gs.suraj.test.1 0 9098 10614 1516 |
| gs.suraj.test.1 2 9098 10614 1516 |
| gs.suraj.test.1 1 9098 10615 1517 |
+-----------------------------------------------------------------+
由于这种滞后，当我在 DC2 中启动同一个消费者时再次读取 4549 条消息，这不应该发生，因为它已经在 DC1 中读取了一个提交，并且镜像制造商已经从 DC1 同步 __consumer_offsets 主题到 DC2
Because of this lag, when I start same consumer in DC2 in reads 4549 messages again, which should not happen as it is already read an commited in DC1 and mirror maker have sync __consumer_offsets topic from DC1 to DC2
如果我在这里遗漏了什么，请告诉我.
Please let me know if I am missing anything in here.
推荐答案
如果您使用的是 Mirror Maker 2.0，他们明确说明不支持仅一次的动机:
If you are using Mirror Maker 2.0 they say explicitly on the motivation that there is no support for exactly-once:
但他们打算在未来这样做.
But they intend to do it in the future.
这篇关于Kafka Mirror Maker:同步 __consumer_offsets 主题重复项的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！