(Apache Beam)无法增加执行程序的内存-尽管使用了多个设置，但它固定为1024M

问题描述

我正在Spark上运行apache Beam工作负载.我用32GB的内存初始化了工作进程(使用 -c 2 -m 32G 进行从属运行).Spark Submit将驱动程序内存设置为30g，将执行程序内存设置为16g.但是，执行程序失败，并出现 java.lang.OutOfMemoryError:Java堆空间.

I am running an apache beam workload on Spark. I initialized the workers with 32GB of memory (slave run with -c 2 -m 32G). Spark submit sets driver memory to 30g and executor memory to 16g. However, executors fail with java.lang.OutOfMemoryError: Java heap space.

主gui指示每个执行程序的内存为1024M.另外，我看到所有Java进程都是使用 -Xmx 1024m 启动的.这意味着spark-submit不会将其执行程序设置传播到执行程序.

The master gui indicates that memory per executor is 1024M. In addition, I see that all java processes are launched with -Xmx 1024m. This means spark-submit doesn't propagate it's executor settings to the executors.

管道选项如下:

   --runner PortableRunner \
    --job_endpoint=localhost:8099 \
    --environment_type=PROCESS \
    --environment_config='{"command": "$HOME/beam/sdks/python/container/build/target/launcher/linux_amd64/boot"}'

作业端点是通过默认方式设置的: docker run --rm --network = host --name spark-jobservice apache/beam_spark_job_server:latest --spark-master-url = spark://$ HOSTNAME:7077

Job endpoint is setup in the default way:docker run --rm --network=host --name spark-jobservice apache/beam_spark_job_server:latest --spark-master-url=spark://$HOSTNAME:7077

如何确保设置传播到执行者?

How do I make sure the settings propagate to the executors?

更新:我将conf/spark-defaults.conf设置为

Update:I set conf/spark-defaults.conf to

spark.driver.memory 32g
spark.executor.memory 32g

和conf/spark-env.sh

and conf/spark-env.sh to

SPARK_EXECUTOR_MEMORY=32g

并重新启动了群集并重新启动了所有内容，执行程序的内存仍然限制为1024M

and restarted the cluster and relaunched everything, and executor memory is still limited to 1024M

推荐答案

我找到了原因和解决方法.

I found the reason and a workaround.

jobserver容器在内部运行其自己的spark分布，因此在本地计算机上的spark分布中配置的设置无效.

The jobserver container is running internally its own spark distribution, so the settings configured in the spark distribution on your local machine have no effect.

因此，解决方案是更改jobserver容器中的配置，例如通过在启动它时传递相应的环境变量:

The solution is thus to change the configuration in the jobserver container, for instance by passing the corresponding environment variable when launching it:

docker run -e SPARK_EXECUTOR_MEMORY=32g --rm --network=host --name spark-jobservice apache/beam_spark_job_server:latest --spark-master-url=spark://$HOSTNAME:7077

这篇关于(Apache Beam)无法增加执行程序的内存-尽管使用了多个设置，但它固定为1024M的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！