连接到在 Docker 中运行的 Kafka

本文介绍了连接到在 Docker 中运行的 Kafka的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！ 问题描述 29岁程序员，3月因学历无情被辞！ 我在本地机器上设置了一个单节点的 Kafka Docker 容器，就像 Confluent 文档(步骤 2-3).此外，我还公开了 Zookeeper 的 2181 端口和 Kafka 的 9092 端口，以便我能够从本地机器上运行的客户端连接到它们:$ docker run -d \-p 2181:2181 \--net=汇合\--name=动物园管理员\-e ZOOKEEPER_CLIENT_PORT=2181 \confluentinc/cp-zookeeper:4.1.0$ docker run -d \--net=汇合\--name=kafka \-p 9092:9092 \-e KAFKA_ZOOKEEPER_CONNECT=动物园管理员:2181 \-e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092 \-e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \confluentinc/cp-kafka:4.1.0问题:当我尝试从主机连接到Kafka时，连接失败，因为它无法解析地址:kafka:9092.这是我的 Java 代码:Properties props = new Properties();props.put("bootstrap.servers", "localhost:9092");props.put("client.id", "KafkaExampleProducer");props.put("key.serializer", LongSerializer.class.getName());props.put("value.serializer", StringSerializer.class.getName());KafkaProducer生产者 = 新的 KafkaProducer<>(props);ProducerRecordrecord = new ProducerRecord("foo", 1L, "Test 1");生产者.send(记录).get();生产者.flush();例外:java.io.IOException:无法解析地址:kafka:9092在 org.apache.kafka.common.network.Selector.doConnect(Selector.java:235) ~[kafka-clients-2.0.0.jar:na]在 org.apache.kafka.common.network.Selector.connect(Selector.java:214) ~[kafka-clients-2.0.0.jar:na]在 org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:864) [kafka-clients-2.0.0.jar:na]在 org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:265) [kafka-clients-2.0.0.jar:na]在 org.apache.kafka.clients.producer.internals.Sender.sendProducerData(Sender.java:266) [kafka-clients-2.0.0.jar:na]在 org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:238) [kafka-clients-2.0.0.jar:na]在 org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:176) [kafka-clients-2.0.0.jar:na]在 java.lang.Thread.run(Thread.java:748) [na:1.8.0_144]引起:java.nio.channels.UnresolvedAddressException:null在 sun.nio.ch.Net.checkAddress(Net.java:101) ~[na:1.8.0_144]在 sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622) ~[na:1.8.0_144]在 org.apache.kafka.common.network.Selector.doConnect(Selector.java:233) ~[kafka-clients-2.0.0.jar:na]... 省略了 7 个常用帧问题: 如何连接到运行在 Docker 中的 Kafka?我的代码是从主机运行的，而不是 Docker.注意:我知道理论上我可以玩弄 DNS 设置和 /etc/hosts 但这是一种解决方法 - 它不应该是那样的.还有类似的问题here，但它基于ches/kafka图像.我使用基于 confluentinc 的图像，这是不一样的. 解决方案 免责声明tl;dr - 归根结底，都是在容器中运行的 Apache Kafka.您只依赖于它是如何配置的.以及哪些变量使它如此.以下使用 confluentinc docker 镜像，而不是 wurstmeister/kafka，尽管 有类似的配置，没试过.如果使用该图片，请阅读他们的 Connectivity wiki. 不反对 wurstmeister 图像，但它是社区维护的，不是建立在自动 CI/CD 版本中的...... Bitnami 的同样简约，维护得更好.对于 bitnami Kafka 图像，参考他们的自述文件debezium/kafka 关于它的文档在此处提及.注意:不推荐使用通告的主机和端口设置.广告听众涵盖了两者.与 Confluent 容器类似，Debezium 可以使用 KAFKA_ 前缀代理设置来更新其属性.spotify/kafka 已弃用且已过时.fast-data-dev 或 lensesio/box 非常适合多合一的解决方案，但如果你只想要 Kafka有关补充阅读、全功能docker-compose 和网络图，请参阅这个博客或 @rmoff 的这个博客答案Confluent 快速入门 (Docker) 文档 假定所有产品消费请求将在 Docker 网络内.您可以通过在使用 Docker 网桥的自己的容器中运行您的 Kafka 客户端代码来解决连接到 kafka:9092 的问题，但否则您需要添加更多环境用于在外部公开容器的变量，同时仍使其在 Docker 网络中工作.首先添加一个PLAINTEXT_HOST:PLAINTEXT的协议映射，它将监听协议映射到一个Kafka协议密钥:KAFKA_LISTENER_SECURITY_PROTOCOL_MAP值:PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT然后在不同的端口上设置两个通告的侦听器.(kafka 此处指的是 docker 容器名称；它也可能被命名为 broker，因此请仔细检查您的服务 + 主机名).注意协议匹配上面映射的右侧值密钥:KAFKA_ADVERTISED_LISTENERS值:PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092运行容器时，为主机端口映射添加-p 29092:29092tl;dr(使用以上设置)如果仍然无法正常工作，可以将 KAFKA_LISTENERS 设置为包含 ://0.0.0.0: 其中两个选项匹配广告设置和 Docker 转发端口客户端在同一台机器上，不在容器中发布 localhost 和关联的端口将使您能够像预期的那样连接到容器外部.换句话说，当在 Docker 网络(包括您可能在本地安装的 CLI 工具)外部运行任何 Kafka 客户端时，使用 localhost:29092 作为引导服务器和 localhost:2181 用于 Zookeeper(需要 Docker 端口转发)另一台机器上的客户端如果尝试从外部服务器连接，您需要公布主机的外部主机名/IP以及/代替本地主机.简单地使用端口转发来通告 localhost 是行不通的，因为 Kafka 协议仍将继续通告您配置的侦听器.如果不在同一个本地网络中，此设置需要 Docker 端口转发和路由器端口转发，例如，您的容器在云中运行，并且您想从本地机器与其交互.容器中的客户端，在同一主机上这是最不容易出错的配置；您可以直接使用 DNS 服务名称.在 Docker 网络中运行应用程序时，使用 kafka:9092(请参阅上面宣传的 PLAINTEXT 侦听器配置)作为引导服务器和 zookeeper:2181 用于 Zookeeper，就像任何其他 Docker 服务通信一样(不需要任何端口转发)如果你使用单独的docker run命令，或者Compose文件，你需要手动定义一个共享的network参见完整 Confluent 堆栈的示例 Compose 文件相关问题从 Docker (ksqlDB) 连接到主机上的 Kafka附录对于对 Kubernetes 部署感兴趣的任何人:https://operatorhub.io/?关键字=KafkaI setup a single node Kafka Docker container on my local machine like it is described in the Confluent documentation (steps 2-3).In addition, I also exposed Zookeeper's port 2181 and Kafka's port 9092 so that I'll be able to connect to them from a client running on local machine:$ docker run -d \ -p 2181:2181 \ --net=confluent \ --name=zookeeper \ -e ZOOKEEPER_CLIENT_PORT=2181 \ confluentinc/cp-zookeeper:4.1.0$ docker run -d \ --net=confluent \ --name=kafka \ -p 9092:9092 \ -e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 \ -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092 \ -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \ confluentinc/cp-kafka:4.1.0Problem: When I try to connect to Kafka from the host machine, the connection fails because it can't resolve address: kafka:9092.Here is my Java code:Properties props = new Properties();props.put("bootstrap.servers", "localhost:9092");props.put("client.id", "KafkaExampleProducer");props.put("key.serializer", LongSerializer.class.getName());props.put("value.serializer", StringSerializer.class.getName());KafkaProducer<Long, String> producer = new KafkaProducer<>(props);ProducerRecord<Long, String> record = new ProducerRecord<>("foo", 1L, "Test 1");producer.send(record).get();producer.flush();The exception:java.io.IOException: Can't resolve address: kafka:9092 at org.apache.kafka.common.network.Selector.doConnect(Selector.java:235) ~[kafka-clients-2.0.0.jar:na] at org.apache.kafka.common.network.Selector.connect(Selector.java:214) ~[kafka-clients-2.0.0.jar:na] at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:864) [kafka-clients-2.0.0.jar:na] at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:265) [kafka-clients-2.0.0.jar:na] at org.apache.kafka.clients.producer.internals.Sender.sendProducerData(Sender.java:266) [kafka-clients-2.0.0.jar:na] at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:238) [kafka-clients-2.0.0.jar:na] at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:176) [kafka-clients-2.0.0.jar:na] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_144]Caused by: java.nio.channels.UnresolvedAddressException: null at sun.nio.ch.Net.checkAddress(Net.java:101) ~[na:1.8.0_144] at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622) ~[na:1.8.0_144] at org.apache.kafka.common.network.Selector.doConnect(Selector.java:233) ~[kafka-clients-2.0.0.jar:na] ... 7 common frames omittedQuestion: How to connect to Kafka running in Docker? My code is running from host machine, not Docker.Note: I know that I could theoretically play around with DNS setup and /etc/hosts but it is a workaround - it shouldn't be like that.There is also similar question here, however it is based on ches/kafka image. I use confluentinc based image which is not the same. 解决方案 Disclaimertl;dr - At the end of the day, it's all the same Apache Kafka running in a container. You're just dependent on how it is configured. And which variables make it so.The following uses confluentinc docker images, not wurstmeister/kafka, although there is a similar configuration, I have not tried it. If using that image, read their Connectivity wiki. Nothing against the wurstmeister image, but it's community maintained, not built in an automated CI/CD release... Bitnami ones are similarly minimalistic and are more well maintained. For bitnami Kafka images, refer their READMEdebezium/kafka docs on it are mentioned here.Note: advertised host and port settings are deprecated. Advertised listeners covers both. Similar to the Confluent containers, Debezium can use KAFKA_ prefixed broker settings to update its properties.spotify/kafka is deprecated and outdated.fast-data-dev or lensesio/box are great for an all in one solution, but are bloated if you only want KafkaFor supplemental reading, a fully-functional docker-compose, and network diagrams, see this blogor this blog by @rmoffAnswerThe Confluent quickstart (Docker) document assumes all produce and consume requests will be within the Docker network.You could fix the problem of connecting to kafka:9092 by running your Kafka client code within its own container as that uses the Docker network bridge, but otherwise you'll need to add some more environment variables for exposing the container externally, while still having it work within the Docker network.First add a protocol mapping of PLAINTEXT_HOST:PLAINTEXT that will map the listener protocol to a Kafka protocolKey: KAFKA_LISTENER_SECURITY_PROTOCOL_MAPValue: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXTThen setup two advertised listeners on different ports. (kafka here refers to the docker container name; it might also be named broker, so double check your service + hostnames). Notice the protocols match the right side values of the mappings aboveKey: KAFKA_ADVERTISED_LISTENERSValue: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092When running the container, add -p 29092:29092 for the host port mappingtl;dr(with the above settings)And if something still doesn't work, KAFKA_LISTENERS can be set to include <PROTOCOL>://0.0.0.0:<PORT> where both options match the advertised setting and Docker-forwarded portClient on same machine, not in a containerAdvertising localhost and the associated port will let you connect outside of the container, as you'd expect.In other words, when running any Kafka Client outside the Docker network (including CLI tools you might have installed locally), use localhost:29092 for bootstrap servers and localhost:2181 for Zookeeper (requires Docker port forwarding)Client on another machineIf trying to connect from an external server, you'll need to advertise the external hostname/ip of the host as well as/in place of localhost.Simply advertising localhost with a port forward will not work because Kafka protocol will still continue to advertise the listeners you've configured.This setup requires Docker port forwarding and router port forwarding if not in the same local network, for example, your container is running in the cloud and you want to interact with it from your local machine.Client in a container, on the same hostThis is the least error-prone configuration; you can use DNS service names directly.When running an app in the Docker network, use kafka:9092 (see advertised PLAINTEXT listener config above) for bootstrap servers and zookeeper:2181 for Zookeeper, just like any other Docker service communication (doesn't require any port forwarding)If you use separate docker run commands, or Compose files, you need to define a shared network manuallySee the example Compose file for the full Confluent stackRelated questionConnect to Kafka on host from Docker (ksqlDB)AppendixFor anyone interested in Kubernetes deployments: https://operatorhub.io/?keyword=Kafka 这篇关于连接到在 Docker 中运行的 Kafka的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！ 上岸，阿里云！