问题描述
在具有YARN的Spark作业中，spark.yarn.executor.memoryOverhead的值应该分配给App还是仅分配给最大值?
The value of spark.yarn.executor.memoryOverhead in a Spark job with YARN should be allocated to App or just the max value?
推荐答案
spark.yarn.executor.memoryOverhead
仅仅是最大值.目标是将RDHEAD和数据帧所使用的OVERHEAD计算为实际执行程序内存的百分比
Is just the max value .The goal is to calculate OVERHEAD as a percentage of real executor memory, as used by RDDs and DataFrames
--executor-memory/spark.executor.memory
控制执行程序堆的大小，但是JVM也可以在堆外使用一些内存，例如，用于内联的String和直接字节缓冲区.
controls the executor heap size, but JVMs can also use some memory off heap, for example for interned Strings and direct byte buffers.
spark.yarn.executor.memoryOverhead的值属性被添加到执行者内存中，以确定每个执行者对YARN的完整内存请求.默认值为max(executorMemory * 0.10，最小为384).
The value of the spark.yarn.executor.memoryOverheadproperty is added to the executor memory to determine the full memory request to YARN for each executor. It defaults to max(executorMemory * 0.10, with minimum of 384).
执行者将使用基于spark.executor.memory属性的内存分配以及由spark.yarn.executor.memoryOverhead
The executors will use a memory allocation based on the property of spark.executor.memoryplus an overhead defined by spark.yarn.executor.memoryOverhead
这篇关于值"spark.yarn.executor.memoryOverhead"设置为"spark.yarn.executor.memoryOverhead".环境?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！