从Spark Streaming读取Kafka SSL客户端信任库文件时出错

问题描述

我有一个从Kafka阅读的Spark流媒体应用程序.我正在从EMR运行它.最近，我实现了Kafka SSL.我正在创建Kafka客户端，如下所示.在尝试读取信任库文件时，运行该应用程序时出现一个奇怪的错误.错误是:

I have a Spark streaming application reading from Kafka. I am running it from EMR. Recently I implemented Kafka SSL. I am creating the Kafka client as shown below. I am getting a strange error running the application when it tries to read the truststore file.Error is:

- Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: java.io.FileNotFoundException: /tmp/kafka.client.truststore.jks (No such file or directory)

是什么原因导致此问题?

What is causing this issue?

DataStreamReader df = session.readStream()
    .format("kafka")
    .option("kafka.bootstrap.servers",kafka_server)
    .option("subscribe", kafka_topic)
    .option("failOnDataLoss", "false")
    .option("group.id", kafka_group)
    .option("kafka.security.protocol","SSL")
    .option("kafka.ssl.truststore.location","/tmp/kafka.client.truststore.jks")
    .option("kafka.ssl.truststore.password","clientpass");

推荐答案

该文件必须存在于集群的执行者节点上，而不仅存在于您从

That file must exist on the executor nodes of the cluster, not just on the machine you submit the code from

否则，您必须传递--files选项以进行火花提交，以将它们从计算机传递给驱动程序

Otherwise, you must pass the --files option to spark-submit to pass them from your machine to the driver

还请注意:结构化流中不允许使用group.id

Also note: group.id isn't allowed in structured streaming

这篇关于从Spark Streaming读取Kafka SSL客户端信任库文件时出错的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！