问题描述
我在本地 kafka 集群中创建了一个主题,有 3 个服务器/代理通过从我的 kafka 安装目录运行以下内容
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 2 --partitions 2 --topic user-activity-tracking-pipeline
一切正常,因为我能够从我的主题生成和消费消息.重新启动机器后,我通过在终端中运行以下命令从 kafka 安装目录启动了捆绑的 zookeeper
bin/zookeeper-server-start.sh config/zookeeper.properties
通过在终端从 kafka 安装目录执行以下命令启动了属于集群的 3 个服务器
env JMX_PORT=10001 bin/kafka-server-start.sh config/server1.properties环境 JMX_PORT=10002 bin/kafka-server-start.sh config/server2.properties环境 JMX_PORT=10003 bin/kafka-server-start.sh config/server3.properties
现在,当我通过在 kafka 安装目录的终端中运行以下命令来列出可用主题时,
bin/kafka-topics.sh --zookeeper localhost:2181 --list
结果为空!
这里是相关的服务器 1 配置条目.服务器 2 和服务器 3 的值非常相似
broker.id=1听众=PLAINTEXT://:9093num.network.threads=3num.io.threads=8socket.send.buffer.bytes=102400socket.receive.buffer.bytes=102400socket.request.max.bytes=104857600log.dirs=/tmp/kafka-logs-broker-1num.partitions=2num.recovery.threads.per.data.dir=1log.retention.hours=168log.segment.bytes=1073741824log.retention.check.interval.ms=300000log.cleaner.enable=falsezookeeper.connect=本地主机:2181zookeeper.connection.timeout.ms=6000
我确实注意到重启后的日志文件,所以没有清理任何东西
/tmp/kafka-logs-broker-1/tmp/kafka-logs-broker-2/tmp/kafka-logs-broker-3
我想知道为什么我尝试列出之前创建的主题 user-activity-tracking-pipeline" 不再存在?
kafka-topics.sh 实际上在底层使用 zookeeper 数据来回答查询.理由是单个经纪人通常无法获得足够的信息来完整地描述主题.
如果您在重启过程中丢失了(我怀疑您确实这样做了,因为您提到了新的 zookeeper 启动)zookeeper 数据,那么 kafka-topics 现在完全失明,无法看到以前的 kafka 数据.
检查发生了什么的最好方法是在查询时实际执行 kafka 正在执行的操作!启动你的zookeeper客户端(就像做./zkCli.sh
一样简单,然后输入ls/brokers/topics
.如果它是空的,你的ZK数据就会丢失.>
I created a topic in my local kafka cluster with 3 servers / brokersby running the following from my kafka installation directory
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 2 --partitions 2 --topic user-activity-tracking-pipeline
Everything worked fine as I was able to produce and consumer messages from my topic. After restarting my machine, I started bundled zookeeper from kafka installation directory by running the following in the terminal
bin/zookeeper-server-start.sh config/zookeeper.properties
Started 3 servers belonging to the cluster by executing the following in terminal from kafka installation directory
env JMX_PORT=10001 bin/kafka-server-start.sh config/server1.properties
env JMX_PORT=10002 bin/kafka-server-start.sh config/server2.properties
env JMX_PORT=10003 bin/kafka-server-start.sh config/server3.properties
Now, when I list available topics by running the following in terminal from kafka installation directory,
bin/kafka-topics.sh --zookeeper localhost:2181 --list
result is empty!
Here are the relevant server 1 configuration entries. The values for server 2 and server 3 are quite similar
broker.id=1
listeners=PLAINTEXT://:9093
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs-broker-1
num.partitions=2
num.recovery.threads.per.data.dir=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
log.cleaner.enable=false
zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=6000
I do notice log files under after restart so nothing was cleaned up
/tmp/kafka-logs-broker-1
/tmp/kafka-logs-broker-2
/tmp/kafka-logs-broker-3
I am wondering why the previously created topic "user-activity-tracking-pipeline" doesn't exist any more when I try to list it?
kafka-topics.sh actually uses zookeeper data under the hood to answer the query. The rationale being that a single broker generally can't have enough information by itself to describe topics completely.
If you lost (which I suspect you did, since you mention a new zookeeper start) zookeeper data during your restart process, kafka-topics is now totally blind and can't see former kafka data.
The best way to check what's happening is to actually do what kafka is doing when you query it ! Launch your zookeeper client (it's as simple as doing ./zkCli.sh
, and type ls /brokers/topics
. If it's empty, your ZK data is lost.
这篇关于重启后Kafka主题不再存在的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!