Debezium Postgres Kafka Connector 心跳未提交 LSN

本文介绍了Debezium Postgres Kafka Connector 心跳未提交 LSN的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！

问题描述

我在 AWS RDS 上有一个 Postgres Db 和一个在桌子上监听的 kafka 连接连接器 (Debezium Postgres).连接器的配置:

I have a Postgres Db on AWS RDS and a kafka connect connector (Debezium Postgres) listening on a table. The configuration of the connector:

{
  "name": "my-connector",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.dbname": "my_db",
    "database.user": "my_user",
    "max.queue.size": "32000",
    "slot.name": "my_slot",
    "tasks.max": "1",
    "publication.name": "my_publication",
    "database.server.name": "postgres",
    "heartbeat.interval.ms": "1000",
    "database.port": "my_port",
    "include.schema.changes": "false",
    "plugin.name": "pgoutput",
    "table.whitelist": "public.my_table",
    "tombstones.on.delete": "false",
    "database.hostname": "my_host",
    "database.password": "my_password",
    "name": "my-connector",
    "max.batch.size": "10000",
    "database.whitelist": "my_db",
    "snapshot.mode": "never"
  },
  "tasks": [
    {
      "connector": "my-connector",
      "task": 0
    }
  ],
  "type": "source"
}

该表的更新频率不如其他表，这最初导致复制延迟如下:

The table is not updated as frequently as other tables, which initially led to replication lag like this:

SELECT slot_name,
  pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) as replicationSlotLag,
  pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn)) as confirmedLag,
  active
FROM pg_replication_slots;
           slot_name           | replicationslotlag | confirmedlag | active
-------------------------------+--------------------+--------------+--------
 my_slot                       | 1664 MB            | 1664 MB      | t

它会变得如此之大，以至于可能会耗尽所有磁盘空间.

It would get so large that it would threaten using up all disk space.

我添加了一个心跳，如果我登录到 kafka 代理并设置这样的控制台使用者: ./kafka-console-consumer.sh --bootstrap-server my.broker.address:9092 --topic __debezium-heartbeat.postgres --from-beginning --consumer.config=/etc/kafka/consumer.properties它会转储所有的心跳消息，然后每 1000 毫秒显示一个新的.

I added a heartbeat, and if I log onto a kafka broker and set up a console consumer like this: ./kafka-console-consumer.sh --bootstrap-server my.broker.address:9092 --topic __debezium-heartbeat.postgres --from-beginning --consumer.config=/etc/kafka/consumer.properties It would dump out all the heartbeat messages, and then show a new one every 1000ms.

然而，插槽的大小仍在增长.如果我执行诸如在表中插入一条虚拟记录之类的操作，它会将插槽设置回一个小的延迟，这样就可以了.

However, the size of the slots still grows and grows. If I do something like insert a dummy record into the table, it sets the slot back to a small lag, so that works.

不过，我想用心跳来做这件事.我不想插入定期消息，因为这听起来会增加复杂性.为什么心跳没有减少插槽大小?

I would like to do this with a heartbeat though. I do not want to insert periodic messages since it sounds like it would add complexity. Why is the heartbeat not reducing the slot size?

it

Debezium Postgres Kafka Connector 心跳未提交 LSN

问题描述

推荐答案