本文介绍了从火花流中的kafka消息中提取时间戳?的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

尝试从 kafka 源中读取.我想从收到的消息中提取时间戳以进行结构化火花流.卡夫卡(版本 0.10.0.0)火花流(2.0.1版)

解决方案
spark.read.format("卡夫卡").option("kafka.bootstrap.servers", "your.server.com:9092").option("订阅", "你的主题").加载().select($"timestamp", $"value")

字段时间戳"就是您要查找的内容.类型 - java.sql.Timestamp.确保您连接到 0.10 Kafka 服务器.早期版本中没有时间戳.此处描述的完整字段列表 - http://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html#creating-a-kafka-source-for-batch-queries>

Trying to read from kafka source. I want to extract timestamp from message received to do structured spark streaming.kafka(version 0.10.0.0)spark streaming(version 2.0.1)

解决方案
spark.read
  .format("kafka")
  .option("kafka.bootstrap.servers", "your.server.com:9092")
  .option("subscribe", "your-topic")
  .load()
  .select($"timestamp", $"value")

Field "timestamp" is what you are looking for. Type - java.sql.Timestamp. Make sure that you are connecting to 0.10 Kafka server. There is no timestamp in earlier versions.Full list of fields described here - http://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html#creating-a-kafka-source-for-batch-queries

这篇关于从火花流中的kafka消息中提取时间戳?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!

09-05 16:03
查看更多