问题描述
运行spark-shell时，它将创建文件derby.log和文件夹metastore_db.如何配置spark将其放置在其他位置?
When running spark-shell it creates a file derby.log and a folder metastore_db. How do I configure spark to put these somewhere else?
对于derby日志，我已经尝试摆脱derby.log 一样spark-shell --driver-memory 10g --conf "-spark.driver.extraJavaOptions=Dderby.stream.info.file=/dev/null"具有几个不同的属性，但spark会忽略它们.
For derby log I've tried Getting rid of derby.log like so spark-shell --driver-memory 10g --conf "-spark.driver.extraJavaOptions=Dderby.stream.info.file=/dev/null" with a couple of different properties but spark ignores them.
有人知道如何摆脱它们或为它们指定默认目录吗?
Does anyone know how to get rid of these or specify a default directory for them?
推荐答案
自Spark 2.0.0起，不再使用hive.metastore.warehouse.dir，参阅文档.
The use of the hive.metastore.warehouse.dir is deprecated since Spark 2.0.0, see the docs.
此答案所暗示的是，metastore_db目录和derby.log文件的真正罪魁祸首是在每个工作子目录中创建的是derby.system.home属性，默认为..
As hinted by this answer, the real culprit for both the metastore_db directory and the derby.log file being created in every working subdirectory is the derby.system.home property defaulting to ..
因此，可以通过将以下行添加到spark-defaults.conf来指定两者的默认位置:
Thus, a default location for both can be specified by adding the following line to spark-defaults.conf:
spark.driver.extraJavaOptions -Dderby.system.home=/tmp/derby
其中/tmp/derby可以替换为您选择的目录.
where /tmp/derby can be replaced by the directory of your choice.
这篇关于如何从Spark Shell摆脱derby.log，metastore_db的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！