问题描述
我正在尝试从 Spark Streaming 访问流式推文.
I am trying to access the streaming tweets from Spark Streaming.
这是软件配置.
Ubuntu 14.04.2 LTS
scala -version
scala -version
Scala 代码运行器版本 2.11.7 -- 版权所有 2002-2013，LAMP/EPFL
spark-submit --version
spark-submit --version
Spark 1.6.0 版
以下是代码.
object PrintTweets
{
def main(args: Array[String]) {
// Configure Twitter credentials using twitter.txt
setupTwitter()
// Set up a Spark streaming context named "PrintTweets" that runs locally using
// all CPU cores and one-second batches of data
val ssc = new StreamingContext("local[*]", "PrintTweets", Seconds(1))
// Get rid of log spam (should be called after the context is set up)
setupLogging()
// Create a DStream from Twitter using our streaming context
val tweets = TwitterUtils.createStream(ssc, None)
// Now extract the text of each status update into RDD's using map()
val statuses = tweets.map(status => status.getText())
// Print out the first ten
statuses.print()
// Kick it all off
ssc.start()
ssc.awaitTermination()
}
}
Utilities.scala
Utilities.scala
object Utilities {
/** Makes sure only ERROR messages get logged to avoid log spam. */
def setupLogging() = {
import org.apache.log4j.{Level, Logger}
val rootLogger = Logger.getRootLogger()
rootLogger.setLevel(Level.ERROR)
}
/** Configures Twitter service credentials using twiter.txt in the main workspace directory */
def setupTwitter() = {
import scala.io.Source
for (line <- Source.fromFile("./data/twitter.txt").getLines) {
val fields = line.split(" ")
if (fields.length == 2) {
System.setProperty("twitter4j.oauth." + fields(0), fields(1))
}
}
}
}
问题:
因为它需要 twitter4j 库，所以我添加了
eclipse 中的 twitter4j-core-4.0.4、twitter4j-stream-4.0.4 作为外部 jars 构建路径.
Since it needs the twitter4j library, i have added
twitter4j-core-4.0.4, twitter4j-stream-4.0.4 in eclipse build path as external jars.
然后我运行了程序，它没有抛出任何错误.但是推文没有出现在控制台中.它是空的.
Then i ran the program, it didnt throw any error. But the tweets not appearing in console. It were empty.
所以我看到一些论坛并将 twitter4j 降级到 3.0.3.同样在 Eclipse 中，我在 Build Path 窗口中选择了 Scala 2.10 Library 容器.
So i see some forums and downgraded twitter4j to 3.0.3. Also in Eclipse i chosen Scala 2.10 Library container in Build Path window.
在那之后我得到了 java.lang.NoSuchMethodError 运行时错误.
After that i got java.lang.NoSuchMethodError run-time error.
16/05/14 11:46:01 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.NoSuchMethodError: twitter4j.TwitterStream.addListener(Ltwitter4j/StreamListener;)V
at org.apache.spark.streaming.twitter.TwitterReceiver.onStart(TwitterInputDStream.scala:72)
at org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:148)
at org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:130)
at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$9.apply(ReceiverTracker.scala:575)
at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$9.apply(ReceiverTracker.scala:565)
at org.apache.spark.SparkContext$$anonfun$37.apply(SparkContext.scala:1992)
at org.apache.spark.SparkContext$$anonfun$37.apply(SparkContext.scala:1992)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
请帮我解决这个问题.最初我通过使用 Scala 2.11 构建安装了 spark.是不是这个问题.我是否需要卸载所有内容并重新安装 Scala 2.10，然后安装 Spark 预编译包.
Please help me to resolve this. Initially i have installed spark by built using Scala 2.11. Is that the problem. Do i need uninstall everything and re-install Scala 2.10, then Spark pre-compiled package.
或者除了 Scala 2.11，我的系统中是否需要安装 Scala 2.10?
Or apart from Scala 2.11, do i need to have Scala 2.10 in my system?
推荐答案
上述异常似乎是spark version 1.6.0和twitter4j 3.0.3 version不兼容造成的.
twitter4j.TwitterStream 有方法addListener 采用 twitter4j.StreamListener 的实例.
twitter4j.TwitterStream which is being passed in the onStart method of org.apache.spark.streaming.twitter.TwitterReceiver, has method addListener which takes instance of twitter4j.StreamListener.
twitter4j 3.0.3 版本 没有方法 twitter4j.TwitterStream.addListener(StreamListener)，而是有几个其他的 addListener 方法，取StreamListener的子类.
twitter4j 3.0.3 version has no method twitter4j.TwitterStream.addListener(StreamListener), instead it has few other addListener methods, which take the subclass of StreamListener.
twitter4j 4.0.4 version 有所需的方法，所以这就是这个库没有错误的原因.所以改成twitter4j 3.0.3 版本并不能解决问题.
twitter4j 4.0.4 version has the desired method, so that's why no error comes with this library. So changing to twitter4j 3.0.3 version will not solve the problem.
问题出在其他地方.
这篇关于Spark Streaming - java.lang.NoSuchMethodError 错误的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！