问题描述
我在从 spark master url 运行应用程序时遇到一个奇怪的问题,其中 UI 无限期地报告WAITING"的STATE",因为 0 个内核显示在 RUNNING APPLICATIONs 表下,无论我配置什么核心计数.
I am having a strange issue with running an application off of the spark master url where the UI is reporting a "STATE" of "WAITING" indefinitely as 0 cores are showing up under the RUNNING APPLICATIONs table no matter what I configure the core count to be.
我使用以下设置配置了我的应用程序,其中 spark.max.cores = 2 &spark.default.cores = 2 &内存设置为3GB.该机器是具有超过 24 个内核的企业级服务器.
Ive configured my app with the following settings where spark.max.cores = 2 & spark.default.cores = 2 & memory set to 3GB. The machine is an enterprise class server with over 24 cores.
SparkConf conf = new SparkConf()
.setAppName(Properties.getString("SparkAppName"))
.setMaster(Properties.getString("SparkMasterUrl"))
.set("spark.executor.memory", Properties.getString("SparkExecMem"))
.set("spark.cores.max",Properties.getString("SparkCores"))
.set("spark.driver.memory",Properties.getString("SparkDriverMem"))
.set("spark.eventLog.enabled", "true")
.set("spark.deploy.defaultCores",Properties.getString("SparkDefaultCores"));
//Set Spark context
JavaSparkContext sc = new JavaSparkContext(conf);
JavaStreamingContext jssc = new JavaStreamingContext(sc, new Duration(5000));
Spark WebUI 声明使用了零个内核并且无限期地等待没有任务运行.该应用程序还在运行时或内核期间使用 NO MEMORY,并在启动时立即进入等待状态.
Spark WebUI states zero cores used and indefinite wait no tasks running. The application is also using NO MEMORY whatsoever during run time or cores and immediately hits a status of waiting when starting.
Spark-defaults.conf
spark.yarn.max_executor.failures 3
spark.yarn.applicationMaster.waitTries 10
spark.history.kerberos.keytab none
spark.yarn.preserve.staging.files False
spark.yarn.submit.file.replication 3
spark.history.kerberos.principal none
spark.yarn.historyServer.address {removed}.{removed}.com:18080
spark.yarn.scheduler.heartbeat.interval-ms 5000
spark.yarn.queue default
spark.yarn.containerLauncherMaxThreads 25
spark.yarn.driver.memoryOverhead 384
spark.history.ui.port 18080
spark.yarn.services org.apache.spark.deploy.yarn.history.YarnHistoryService
spark.yarn.max.executor.failures 3
spark.driver.extraJavaOptions -Dhdp.version=2.2.6.0-2800
spark.history.provider org.apache.spark.deploy.yarn.history.YarnHistoryProvider
spark.yarn.am.extraJavaOptions -Dhdp.version=2.2.6.0-2800
spark.yarn.executor.memoryOverhead 384
提交脚本
spark-submit --class {removed}.{removed}.{removed}.sentiment.MainApp --deploy-mode client/path/to/jar
2/3/2016使用 --master yarn-cluster 运行后,我在纱线日志错误中收到了这个.我还包括了我更新的提交配置
EDITED: 2/3/2016After running with --master yarn-cluster I am receiving this in the yarn logs error. I have also included my updated submit configuration
提交配置
spark-submit --class com.removed.removed.sentiment.MainApp
--master yarn-cluster --supervise
/data04/dev/removed/spark/twitternpi/npi.sentiment-1.0-SNAPSHOT-shaded.jar
--jars /usr/hdp/2.2.6.0-2800/spark/lib/datanucleus-core-3.2.10.jar,/usr/hdp/2.2.6.0-2800/spark/lib/datanucleus-api-jdo-3.2.6.jar,/usr/hdp/2.2.6.0-2800/spark/lib/datanucleus-rdbms-3.2.9.jar,/usr/hdp/2.2.6.0-2800/spark/lib/spark-1.2.1.2.2.6.0-2800-yarn-shuffle.jar,/usr/hdp/2.2.6.0-2800/spark/lib/spark-assembly-1.2.1.2.2.6.0-2800-hadoop2.6.0.2.2.6.0-2800.jar
错误信息
ClassLoaderResolver for class "" gave error on creation : {1}
org.datanucleus.exceptions.NucleusUserException: ClassLoaderResolver for class "" gave error on creation : {1}
at org.datanucleus.NucleusContext.getClassLoaderResolver(NucleusContext.java:1087)
at org.datanucleus.PersistenceConfiguration.validatePropertyValue(PersistenceConfiguration.java:797)
at org.datanucleus.PersistenceConfiguration.setProperty(PersistenceConfiguration.java:714)
at org.datanucleus.PersistenceConfiguration.setPersistenceProperties(PersistenceConfiguration.java:693)
at org.datanucleus.NucleusContext.<init>(NucleusContext.java:273)
at org.datanucleus.NucleusContext.<init>(NucleusContext.java:247)
at org.datanucleus.NucleusContext.<init>(NucleusContext.java:225)
推荐答案
当执行器所需的内存大小由 spark 中的
,比 AWS 节点上的要大.但是由于您只设置了 3.0 GB 作为您的内存,我认为您的情况可能有其他原因.spark.executor.memory
设置时,我遇到了这个问题-defaults.conf
I ran into this problem when the required memory size for the executor, set by spark.executor.memory
in spark-defaults.conf
, is bigger than that on the AWS node. But since you only set 3.0 GB as your memory, I think there might be other causes in your case.
这篇关于即使在应用程序中设置内核时,Spark UI 也显示 0 个内核的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!