问题描述
我有以下问题要解决.你们是否知道允许根据主题前缀设置不同的最小ISR值的机制?例如,使用前缀 app-*
创建的主题的最小ISR因子值设置为2, logs-*
等于1,但 core-* 的值code>为4.
I have following problem to solve. Do You Guys know mechanism that would allow set different min ISR value based on topic prefix? For example topics created with prefix app-*
would have min ISR factor value set to 2, logs-*
equal to 1 but value for core-*
would be 4.
min.insync.replicas.per.topic = topic_name_1:value,topic_name_2:value
看起来几乎是一种解决方案,但不幸的是,当 min.insync.replicas
也出现在 kafka.properties
中.因此,我将探讨是否存在将这两个道具结合在一起的解决方案.来源
min.insync.replicas.per.topic=topic_name_1:value,topic_name_2:value
looks almost as a solution but unfortunately it is shadowed when min.insync.replicas
is also present in kafka.properties
. So I will explore if there is solution to combine those two props.source
使用验证最小ISR值
./kafka-configs.sh --bootstrap-server localhost:9092 \
--entity-type topics --entity-name $TOPIC --describe --all
推荐答案
为了避免使用80年代风格的解决方案(如本答案所示),请查看Dirk的答案.
我只是喜欢bash/sh
对于已经创建的主题,您必须执行分区重新分配操作.
For already created topics, you'd have to execute a partition reassignment operation.
如果您希望控制新创建主题的 ISR
,则可以将 kafka-topics.sh
脚本包装在新脚本中,我们称之为 createTopics.sh
:
If you wish to control the ISR
for newly created topics, you could wrap the kafka-topics.sh
script in a new script, lets call it createTopics.sh
:
#!/bin/bash
isr=0
if [[ $1 = \logs* ]]; then
isr=1
elif [[ $1 = \app* ]]; then
isr=2
elif [[ $1 = \core* ]]; then
isr=4
fi
/bin/kafka-topics.sh --create --topic $1 --replication-factor $isr \
--partitions $3 --bootstrap-server $2
-
$ 1
-主题名称 -
$ 2
-引导服务器 -
$ 3
-分区 $1
- topic name$2
- bootstrap-server$3
- partitions
例如:
./createTopics.sh apps-topic localhost:9092 1 && \
./createTopics.sh core-topic localhost:9092 1
这将创建两个具有各自复制因子的新主题:
This would create two new topics with their respective replication-factor:
-
应用主题
>ISR
= 2 . -
核心主题
>ISR
= 4 .
apps-topic
>ISR
= 2.core-topic
>ISR
= 4.
(最大ISR值,假设所有副本均处于同步状态)
该脚本还将验证新主题的创建,因为它不会创建名称不以以下任何一个开头的主题: logs/app/core .Kafka将显示错误,复制因子必须大于0
.
The script would also validate the creation of new topics, as it wouldn't create any topic which name doesn't start with one of these: logs / app / core. Kafka would show an error, Replication factor must be larger than 0
.
您可以更改此行为,以允许使用不同的主题前缀,例如,通过在脚本开头将默认isr值设置为不同于0的值: isr = 1
.这将导致:
You could change this behaviour in order to allow different topic prefixes, by setting the default isr value at the start of the script different than 0: isr=1
, for example. This would lead to:
#!/bin/bash
isr=1
if [[ $1 = \app* ]]; then
isr=2
elif [[ $1 = \core* ]]; then
isr=4
fi
/bin/kafka-topics.sh --create --topic $1 --replication-factor $isr \
--partitions $3 --bootstrap-server $2
这篇关于Kafka-根据主题前缀设置最小ISR的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!