本文介绍了即使在应用程序中设置内核时,Spark UI 也显示 0 个内核的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

我在从 spark master url 运行应用程序时遇到一个奇怪的问题,其中 UI 无限期地报告WAITING"的STATE",因为 0 个内核显示在 RUNNING APPLICATIONs 表下,无论我配置什么核心计数.

I am having a strange issue with running an application off of the spark master url where the UI is reporting a "STATE" of "WAITING" indefinitely as 0 cores are showing up under the RUNNING APPLICATIONs table no matter what I configure the core count to be.

我使用以下设置配置了我的应用程序,其中 spark.max.cores = 2 &spark.default.cores = 2 &内存设置为3GB.该机器是具有超过 24 个内核的企业级服务器.

Ive configured my app with the following settings where spark.max.cores = 2 & spark.default.cores = 2 & memory set to 3GB. The machine is an enterprise class server with over 24 cores.

        SparkConf conf = new SparkConf()
            .setAppName(Properties.getString("SparkAppName"))
            .setMaster(Properties.getString("SparkMasterUrl"))
            .set("spark.executor.memory", Properties.getString("SparkExecMem"))
            .set("spark.cores.max",Properties.getString("SparkCores"))
            .set("spark.driver.memory",Properties.getString("SparkDriverMem"))
            .set("spark.eventLog.enabled", "true")
            .set("spark.deploy.defaultCores",Properties.getString("SparkDefaultCores"));

    //Set Spark context
    JavaSparkContext sc = new JavaSparkContext(conf);
    JavaStreamingContext jssc = new JavaStreamingContext(sc, new Duration(5000));

Spark WebUI 声明使用了零个内核并且无限期地等待没有任务运行.该应用程序还在运行时或内核期间使用 NO MEMORY,并在启动时立即进入等待状态.

Spark WebUI states zero cores used and indefinite wait no tasks running. The application is also using NO MEMORY whatsoever during run time or cores and immediately hits a status of waiting when starting.

Spark-defaults.conf
spark.yarn.max_executor.failures         3
spark.yarn.applicationMaster.waitTries   10
spark.history.kerberos.keytab    none
spark.yarn.preserve.staging.files        False
spark.yarn.submit.file.replication       3
spark.history.kerberos.principal         none
spark.yarn.historyServer.address         {removed}.{removed}.com:18080
spark.yarn.scheduler.heartbeat.interval-ms       5000
spark.yarn.queue         default
spark.yarn.containerLauncherMaxThreads   25
spark.yarn.driver.memoryOverhead         384
spark.history.ui.port    18080
spark.yarn.services      org.apache.spark.deploy.yarn.history.YarnHistoryService
spark.yarn.max.executor.failures         3
spark.driver.extraJavaOptions     -Dhdp.version=2.2.6.0-2800
spark.history.provider   org.apache.spark.deploy.yarn.history.YarnHistoryProvider
spark.yarn.am.extraJavaOptions    -Dhdp.version=2.2.6.0-2800
spark.yarn.executor.memoryOverhead       384

提交脚本

spark-submit --class {removed}.{removed}.{removed}.sentiment.MainApp --deploy-mode client/path/to/jar

2/3/2016使用 --master yarn-cluster 运行后,我在纱线日志错误中收到了这个.我还包括了我更新的提交配置

EDITED: 2/3/2016After running with --master yarn-cluster I am receiving this in the yarn logs error. I have also included my updated submit configuration

提交配置

spark-submit --class com.removed.removed.sentiment.MainApp
--master yarn-cluster --supervise
/data04/dev/removed/spark/twitternpi/npi.sentiment-1.0-SNAPSHOT-shaded.jar
--jars /usr/hdp/2.2.6.0-2800/spark/lib/datanucleus-core-3.2.10.jar,/usr/hdp/2.2.6.0-2800/spark/lib/datanucleus-api-jdo-3.2.6.jar,/usr/hdp/2.2.6.0-2800/spark/lib/datanucleus-rdbms-3.2.9.jar,/usr/hdp/2.2.6.0-2800/spark/lib/spark-1.2.1.2.2.6.0-2800-yarn-shuffle.jar,/usr/hdp/2.2.6.0-2800/spark/lib/spark-assembly-1.2.1.2.2.6.0-2800-hadoop2.6.0.2.2.6.0-2800.jar

错误信息

   ClassLoaderResolver for class "" gave error on creation : {1}
org.datanucleus.exceptions.NucleusUserException: ClassLoaderResolver for class "" gave error on creation : {1}
    at org.datanucleus.NucleusContext.getClassLoaderResolver(NucleusContext.java:1087)
    at org.datanucleus.PersistenceConfiguration.validatePropertyValue(PersistenceConfiguration.java:797)
    at org.datanucleus.PersistenceConfiguration.setProperty(PersistenceConfiguration.java:714)
    at org.datanucleus.PersistenceConfiguration.setPersistenceProperties(PersistenceConfiguration.java:693)
    at org.datanucleus.NucleusContext.<init>(NucleusContext.java:273)
    at org.datanucleus.NucleusContext.<init>(NucleusContext.java:247)
    at org.datanucleus.NucleusContext.<init>(NucleusContext.java:225)

推荐答案

当执行器所需的内存大小由 spark 中的 spark.executor.memory 设置时,我遇到了这个问题-defaults.conf,比 AWS 节点上的要大.但是由于您只设置了 3.0 GB 作为您的内存,我认为您的情况可能有其他原因.

I ran into this problem when the required memory size for the executor, set by spark.executor.memory in spark-defaults.conf, is bigger than that on the AWS node. But since you only set 3.0 GB as your memory, I think there might be other causes in your case.

这篇关于即使在应用程序中设置内核时,Spark UI 也显示 0 个内核的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!

08-20 13:14
查看更多