提交带有Spark typesafe配置的应用程序属性文件

问题描述

请，我需要您的帮助，我正在尝试使用typesafe config为我的spark应用程序提交外部配置文件.

Please, I need your help, I'm trying to submit an external configuration file for my spark application using typesafe config.

我正在这样在我的应用程序代码中加载application.conf文件:

I'm loading the application.conf file in my application code like this:

  lazy val conf = ConfigFactory.load()

文件内容

  ingestion{
  process {
    value = "sas"
  }
  sas {
    origin{
      value = "/route"
    }
    destination{
      value = "/route"
    }
    extension{
      value = ".sas7bdat"
    }
    file{
      value = "mytable"
    }
    month{
      value = "201010,201011"
    }
    table{
      value = "tbl"
    }
  }
}

我的火花提交是

spark2-submit --class com.antonio.Main --master yarn --deploy-mode client --driver-memory 10G --driver-cores 8 --executor-memory 13G --executor-cores 4 --num-executors 10 --verbose  --files properties.conf /home/user/ingestion-1.0-SNAPSHOT-jar-with-dependencies.jar --files application.conf

但是由于某些原因，我收到了

But for some reason, I'm receiving

com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'ingestion'

一切看起来都正确配置了??我错过了什么吗?

Everything looks configured correctly ?? Have I missed something.

谢谢

安东尼奥

推荐答案

默认情况下，您的application.conf必须存在于类路径的根目录下，以便ConfigFactory.load()能够找到它.或者，您可以通过系统属性修改在哪里找到application.conf文件.因此，您的选择如下.

Your application.conf by default must be present at the root of classpath for ConfigFactory.load() to find it. Alternatively, you can modify where to find the application.conf file through system properties. Therefore, your options are as follows.

第一种选择是，将作业的根目录添加到classpath中:

First alternative is, add the root directory of the job to classpath:

spark2-submit ... \
    --conf spark.driver.extraClassPath=./  \
    --conf spark.executor.extraClassPath=./  \    // if you need to load config at executors
    ...

保持--files选项不变.请注意，如果您在客户端模式下运行作业，则必须将正确的路径传递到spark.driver.extraClassPath选项的驱动器计算机上application.conf所在的位置.

Keep the --files option as is. Note that if you run your job in the client mode, you must pass the proper path to where application.conf is located on the driver machine to the spark.driver.extraClassPath option.

第二种选择是(我认为这是更好的选择)，您可以使用config.file系统属性来影响ConfigFactory.load()在哪里寻找配置文件:

Second alternative is (and I think this one is superior), you can use the config.file system property to affect where ConfigFactory.load() looks for the config file:

spark2-submit ... \
    --conf spark.driver.extraJavaOptions=-Dconfig.file=./application.conf \
    --conf spark.executor.extraJavaOptions=-Dconfig.file=./application.conf \
    ...

关于在执行程序上加载配置并保留--files选项的说明也适用于此处.

Remarks about loading config on executors and keeping the --files option also apply here.

这篇关于提交带有Spark typesafe配置的应用程序属性文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！