问题描述
我想设置 spark.eventLog.enabled 和 spark.eventLog.dir 在火花提交或开始 - 所有级 - 不要求它在斯卡拉/ JAVA /蟒蛇code启用。
我曾尝试各种事情没有成功：
I would like to set spark.eventLog.enabled and spark.eventLog.dir at the spark-submit or start-all level -- not require it to be enabled in the scala/java/python code. I have tried various things with no success:
spark.eventLog.enabled true
spark.eventLog.dir hdfs://namenode:8021/directory
或
spark.eventLog.enabled true
spark.eventLog.dir file:///some/where
运行火花提交为：
spark-submit --conf "spark.eventLog.enabled=true" --conf "spark.eventLog.dir=file:///tmp/test" --master spark://server:7077 examples/src/main/python/pi.py
与环境变量开始火花：
SPARK_DAEMON_JAVA_OPTS="-Dspark.eventLog.enabled=true -Dspark.history.fs.logDirectory=$sparkHistoryDir -Dspark.history.provider=org.apache.spark.deploy.history.FsHistoryProvider -Dspark.history.fs.cleaner.enabled=true -Dspark.history.fs.cleaner.interval=2d"
和只是为了矫枉过正：
SPARK_HISTORY_OPTS="-Dspark.eventLog.enabled=true -Dspark.history.fs.logDirectory=$sparkHistoryDir -Dspark.history.provider=org.apache.spark.deploy.history.FsHistoryProvider -Dspark.history.fs.cleaner.enabled=true -Dspark.history.fs.cleaner.interval=2d"
在哪里以及如何必须这些东西被设定为获得任意工作历史？
Where and how must these things be set to get history on arbitrary jobs?
推荐答案
我解决了这个问题，但奇怪的是我以前试过这样...
尽管如此，现在它似乎是一个稳定的解决方案：
I solved the problem, yet strangely I had tried this before...All the same, now it seems like a stable solution:
创建日志记录 HDFS 的目录，比如 / EVENTLOGGING
Create a directory in HDFS for logging, say /eventLogging
HDFS DFS -mkdir / EVENTLOGGING
然后火花壳或火花提交（或其他）可以用下面的选项来运行： - 配置spark.eventLog.enabled =真--conf spark.eventLog.dir = HDFS：//＆LT; hdfsNameNodeAddress＆GT;：8020 / EVENTLOGGING
Then spark-shell or spark-submit (or whatever) can be run with the following options: --conf spark.eventLog.enabled=true --conf spark.eventLog.dir=hdfs://<hdfsNameNodeAddress>:8020/eventLogging
如：
火花壳--conf spark.eventLog.enabled =真--conf spark.eventLog.dir = HDFS：//＆LT; hdfsNameNodeAddress＆GT;：8020 / EVENTLOGGING
这篇关于Apache的火花：在设置spark.eventLog.enabled和spark.eventLog.dir提交或启动星火的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！