我正在尝试使用spark的简单工作计数程序,但是当我尝试初始化spark上下文时,该程序失败。
下面是我的代码
conf = new SparkConf(true).
setAppName("WordCount").
setMaster("spark://192.168.0.104:7077");
sc = new JavaSparkContext(conf);
现在我想澄清一下我正在使用Spark版本2.1.1,我的Java代码在Windows 10上,我的服务器在VM盒上运行。
我已禁用VM中的防火墙,并且可以从Windows访问URL http://192.168.0.104:8080/。
但是我在运行代码时低于stacktrace
17/08/06 18:44:15 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://192.168.0.103:4040
17/08/06 18:44:15 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://192.168.0.104:7077...
17/08/06 18:44:15 INFO TransportClientFactory: Successfully created connection to /192.168.0.104:7077 after 41 ms (0 ms spent in bootstraps)
17/08/06 18:44:15 WARN StandaloneAppClient$ClientEndpoint: Failed to connect to master 192.168.0.104:7077
org.apache.spark.SparkException: Exception thrown in awaitResult
at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:77)
at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:75)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)
at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)
at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)
有人可以帮忙吗?
最佳答案
您需要将一些Spark类导入程序。添加以下行:
import org.apache.spark.api.java.JavaSparkContext
import org.apache.spark.api.java.JavaRDD
import org.apache.spark.SparkConf
SparkConf conf = new SparkConf().setAppName("WordCount").setMaster("local");
JavaSparkContext sc = new JavaSparkContext(conf);
关于java - 无法使用Java初始化Spark上下文,我们在Stack Overflow上找到一个类似的问题:https://stackoverflow.com/questions/45532596/