本文介绍了Kafka-根据主题前缀设置最小ISR的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

我有以下问题要解决.你们是否知道允许根据主题前缀设置不同的最小ISR值的机制?例如,使用前缀 app-* 创建的主题的最小ISR因子值设置为2, logs-* 等于1,但 core-* 为4.

I have following problem to solve. Do You Guys know mechanism that would allow set different min ISR value based on topic prefix? For example topics created with prefix app-* would have min ISR factor value set to 2, logs-* equal to 1 but value for core-* would be 4.

min.insync.replicas.per.topic = topic_name_1:value,topic_name_2:value 看起来几乎是一种解决方案,但不幸的是,当 min.insync.replicas 也出现在 kafka.properties 中.因此,我将探讨是否存在将这两个道具结合在一起的解决方案.来源

min.insync.replicas.per.topic=topic_name_1:value,topic_name_2:value looks almost as a solution but unfortunately it is shadowed when min.insync.replicas is also present in kafka.properties. So I will explore if there is solution to combine those two props.source

使用验证最小ISR值

./kafka-configs.sh --bootstrap-server localhost:9092 \
 --entity-type topics --entity-name $TOPIC --describe --all

推荐答案

为了避免使用80年代风格的解决方案(如本答案所示),请查看Dirk的答案.

我只是喜欢bash/sh

对于已经创建的主题,您必须执行分区重新分配操作.

For already created topics, you'd have to execute a partition reassignment operation.

如果您希望控制新创建主题的 ISR ,则可以将 kafka-topics.sh 脚本包装在新脚本中,我们称之为 createTopics.sh :

If you wish to control the ISR for newly created topics, you could wrap the kafka-topics.sh script in a new script, lets call it createTopics.sh:

#!/bin/bash
isr=0

if [[ $1 = \logs* ]]; then
        isr=1
elif  [[ $1 = \app* ]]; then
        isr=2
elif  [[ $1 = \core* ]]; then
        isr=4
fi

/bin/kafka-topics.sh --create --topic $1 --replication-factor $isr \
                     --partitions $3  --bootstrap-server $2

  • $ 1 -主题名称
  • $ 2 -引导服务器
  • $ 3 -分区
    • $1 - topic name
    • $2 - bootstrap-server
    • $3 - partitions
    • 例如:

        ./createTopics.sh apps-topic localhost:9092 1 && \
        ./createTopics.sh core-topic localhost:9092 1
      

      这将创建两个具有各自复制因子的新主题:

      This would create two new topics with their respective replication-factor:

      • 应用主题 > ISR = 2 .
      • 核心主题 > ISR = 4 .
      • apps-topic > ISR = 2.
      • core-topic > ISR = 4.

      (最大ISR值,假设所有副本均处于同步状态)

      该脚本还将验证新主题的创建,因为它不会创建名称不以以下任何一个开头的主题: logs/app/core .Kafka将显示错误,复制因子必须大于0 .

      The script would also validate the creation of new topics, as it wouldn't create any topic which name doesn't start with one of these: logs / app / core. Kafka would show an error, Replication factor must be larger than 0.

      您可以更改此行为,以允许使用不同的主题前缀,例如,通过在脚本开头将默认isr值设置为不同于0的值: isr = 1 .这将导致:

      You could change this behaviour in order to allow different topic prefixes, by setting the default isr value at the start of the script different than 0: isr=1, for example. This would lead to:

      #!/bin/bash
      isr=1
      
      if  [[ $1 = \app* ]]; then
             isr=2
      elif  [[ $1 = \core* ]]; then
             isr=4
      fi
      
      /bin/kafka-topics.sh --create --topic $1 --replication-factor $isr \
                           --partitions $3  --bootstrap-server $2
      

      这篇关于Kafka-根据主题前缀设置最小ISR的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!

09-22 23:38