请参阅 API文档以获取更多详细信息I want just to print the content of streams to console. I wrote the following code but it does not print anything. Anyone can help me to read text file as stream in Spark?? Is there a problem related to Windows system?public static void main(String[] args) throws Exception { SparkConf sparkConf = new SparkConf().setAppName("My app") .setMaster("local[2]") .setSparkHome("C:\\Spark\\spark-1.5.1-bin-hadoop2.6") .set("spark.executor.memory", "2g"); JavaStreamingContext jssc = new JavaStreamingContext(sparkConf, Durations.seconds(2)); JavaDStream<String> dataStream = jssc.textFileStream("C://testStream//copy.csv"); dataStream.print(); jssc.start(); jssc.awaitTermination();}UPDATE: The content of copy.csv is0,0,12,5,00,0,12,5,00,1,2,0,420,0,0,0,2640,0,12,5,0 解决方案 textFileStream is for Monitoring the hadoop Compatible Directories. This operation will watch the provided directory and as you add new files in the provided directory it will read/ stream the data from the newly added files.You cannot read text/ csv files using textFileStream or rather I would say that you do not need streaming in case you are just reading the files.My Suggestion would be to monitor some directory (may be HDFS or local file system) and then add files and capture the content of these new files using textFileStream.May be in your code may be you can replace "C://testStream//copy.csv" with C://testStream" and once your Spark Streaming job is up and running then add file copy.csv to C://testStream folder and see the output on Spark Console.ORmay be you can write another command line Scala/ Java program which read the files and throw the content over the Socket (at a certain PORT#) and next you can leverage socketTextStream for capturing and reading the data. Once you have read the data, you further apply other transformation or output operations.You can also think of leveraging Flume tooRefer to API Documentation for more details 这篇关于在Windows系统中打印流的内容(火花流)的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持! 上岸,阿里云!
08-20 13:16
查看更多