问题描述
- 我已经在Kubernetes集群中部署并运行了Kafka.我正在使用来自docker hub的图像- https://hub.docker.com/r/cloudtrackinc/kubernetes-kafka/
- 我的kubernetes集群中有3个kube节点.我有3个Kafka和3个zookeeper应用程序正在运行,并且有对应于它们运行的服务zoo1,zoo2,zoo3和kafka-1,kafka-2和kafka-3.我可以从kubernetes集群内部发布/使用,但不能从kubernetes集群外部发布/使用,即从不属于kubernetes集群的外部机器发布/使用.
- 我能够从外部计算机访问kube节点-基本上我可以使用name/ip对其执行ping操作.
- 我没有使用任何外部负载平衡器,但是我有一个可以同时解析我的外部计算机和kube节点的DNS.
- 在这种情况下,无法使用NodePort或ExternalIP公开Kafka服务. 最终在
- 设置
KAFKA_ADVERTISED_HOST_NAME
或KAFKA_ADVERTISED_LISTENERS
都不能帮助从kubernetes集群外部访问kafka.
server.properties
中设置ADVERTISED_HOST_NAME
/ADVERTISED_LISTENERS
属性的Kafka RC YML中的- I have Kafka deployed and running in Kubernetes cluster. I am using this image from docker hub - https://hub.docker.com/r/cloudtrackinc/kubernetes-kafka/
- I have 3 kube-nodes in my kubernetes cluster. I have 3 Kafka and 3 zookeeper applications running and I have services zoo1,zoo2,zoo3 and kafka-1, kafka-2 and kafka-3 running corresponding to them. I am able to publish/consume from inside kubernetes cluster but I am not able to publish/consume from outside of kubernetes cluster i.e., from external machine not part of kubernetes cluster.
- I am able to reach the kube-nodes from external machine - basically I can ping them using name/ip.
- I am not using any external load balancer but I have a DNS that can resolve both my external machine and kube-nodes.
- Using NodePort or ExternalIP to expose the Kafka service does not work in this case.
- Setting
KAFKA_ADVERTISED_HOST_NAME
orKAFKA_ADVERTISED_LISTENERS
in Kafka RC YML that ultimately setADVERTISED_HOST_NAME
/ADVERTISED_LISTENERS
properties inserver.properties
either does not help accessing kafka from outside of kubernetes cluster.
请建议我如何从kubernetes集群外部发布/使用.非常感谢!
Please suggest how can I publish/consume from outside of kubernetes cluster. Thanks much!
推荐答案
我在从AWS上的k8s集群外部访问kafka时遇到了同样的问题.我设法通过使用Kafka侦听器功能解决此问题,该功能从0.10.2版本开始支持多个接口.
I had the same problem with accessing kafka from outside of k8s cluster on AWS. I manage to solve this issue by using kafka listeners feature which from version 0.10.2 supports multiple interfaces.
这是我配置kafka容器的方式.
here is how I configured kafka container.
ports:
- containerPort: 9092
- containerPort: 9093
env:
- name: KAFKA_ZOOKEEPER_CONNECT
value: "zookeeper:2181"
- name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
value: "INTERNAL_PLAINTEXT:PLAINTEXT,EXTERNAL_PLAINTEXT:PLAINTEXT"
- name: KAFKA_ADVERTISED_LISTENERS
value: "INTERNAL_PLAINTEXT://kafka-internal-service:9092,EXTERNAL_PLAINTEXT://123.us-east-2.elb.amazonaws.com:9093"
- name: KAFKA_LISTENERS
value: "INTERNAL_PLAINTEXT://0.0.0.0:9092,EXTERNAL_PLAINTEXT://0.0.0.0:9093"
- name: KAFKA_INTER_BROKER_LISTENER_NAME
value: "INTERNAL_PLAINTEXT"
除此之外,我还配置了两个服务.一种用于内部(无头)和一个用于外部(LoadBalancer)通信.
Apart from that I configured two Services. One for internal(Headless) & one for external(LoadBalancer) communication.
希望这可以节省人们的时间.
Hopefully this will save people's time.
这篇关于Kubernetes Cluster中的Kafka-如何从Kubernetes Cluster外部发布/使用消息的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!