问题描述
尝试从 kafka 源中读取.我想从收到的消息中提取时间戳以进行结构化火花流.卡夫卡(版本 0.10.0.0)火花流(2.0.1版)
spark.read.format("卡夫卡").option("kafka.bootstrap.servers", "your.server.com:9092").option("订阅", "你的主题").加载().select($"timestamp", $"value")
字段时间戳"就是您要查找的内容.类型 - java.sql.Timestamp.确保您连接到 0.10 Kafka 服务器.早期版本中没有时间戳.此处描述的完整字段列表 - http://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html#creating-a-kafka-source-for-batch-queries>
Trying to read from kafka source. I want to extract timestamp from message received to do structured spark streaming.kafka(version 0.10.0.0)spark streaming(version 2.0.1)
spark.read
.format("kafka")
.option("kafka.bootstrap.servers", "your.server.com:9092")
.option("subscribe", "your-topic")
.load()
.select($"timestamp", $"value")
Field "timestamp" is what you are looking for. Type - java.sql.Timestamp. Make sure that you are connecting to 0.10 Kafka server. There is no timestamp in earlier versions.Full list of fields described here - http://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html#creating-a-kafka-source-for-batch-queries
这篇关于从火花流中的kafka消息中提取时间戳?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!