我有一个小麻烦，想知道为什么我的Spark工作确切地死了，所以我将在此帖子的底部包含追溯，以便比我更有经验的人可以给我一些见解：)据我所知节点快要死了，因为超出了memoryOverhead。如何从awscli进行设置，以免遇到此问题？
这是我的一些回溯：
16/05/17 20:20:46 WARN TaskSetManager: Lost task 97.0 in stage 3.0 (TID 9937, ip-172-31-14-59.us-west-2.compute.internal): ExecutorLostFailure (executor 9 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
16/05/17 20:20:46 WARN TaskSetManager: Lost task 60.0 in stage 3.0 (TID 9900, ip-172-31-14-59.us-west-2.compute.internal): ExecutorLostFailure (executor 9 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
16/05/17 20:20:46 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
16/05/17 20:20:46 WARN TaskSetManager: Lost task 134.0 in stage 3.0 (TID 9974, ip-172-31-14-59.us-west-2.compute.internal): ExecutorLostFailure (executor 9 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
16/05/17 20:20:46 WARN TaskSetManager: Lost task 23.0 in stage 3.0 (TID 9863, ip-172-31-14-59.us-west-2.compute.internal): ExecutorLostFailure (executor 9 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
16/05/17 20:20:46 INFO YarnClientSchedulerBackend: Asked to remove non-existent executor 9
16/05/17 20:20:46 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
16/05/17 20:20:46 ERROR YarnScheduler: Lost executor 15 on ip-172-31-14-46.us-west-2.compute.internal: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
16/05/17 20:20:46 WARN TaskSetManager: Lost task 88.0 in stage 3.0 (TID 9928, ip-172-31-14-46.us-west-2.compute.internal): ExecutorLostFailure (executor 15 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
16/05/17 20:20:46 WARN TaskSetManager: Lost task 51.0 in stage 3.0 (TID 9891, ip-172-31-14-46.us-west-2.compute.internal): ExecutorLostFailure (executor 15 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
16/05/17 20:20:46 WARN TaskSetManager: Lost task 125.0 in stage 3.0 (TID 9965, ip-172-31-14-46.us-west-2.compute.internal): ExecutorLostFailure (executor 15 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
16/05/17 20:20:46 WARN TaskSetManager: Lost task 14.0 in stage 3.0 (TID 9854, ip-172-31-14-46.us-west-2.compute.internal): ExecutorLostFailure (executor 15 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
16/05/17 20:20:46 INFO YarnClientSchedulerBackend: Asked to remove non-existent executor 15
16/05/17 20:20:46 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
16/05/17 20:20:46 ERROR YarnScheduler: Lost executor 14 on ip-172-31-14-61.us-west-2.compute.internal: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
16/05/17 20:20:46 WARN TaskSetManager: Lost task 85.0 in stage 3.0 (TID 9925, ip-172-31-14-61.us-west-2.compute.internal): ExecutorLostFailure (executor 14 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
16/05/17 20:20:46 WARN TaskSetManager: Lost task 48.0 in stage 3.0 (TID 9888, ip-172-31-14-61.us-west-2.compute.internal): ExecutorLostFailure (executor 14 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
16/05/17 20:20:46 WARN TaskSetManager: Lost task 122.0 in stage 3.0 (TID 9962, ip-172-31-14-61.us-west-2.compute.internal): ExecutorLostFailure (executor 14 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
16/05/17 20:20:46 WARN TaskSetManager: Lost task 11.0 in stage 3.0 (TID 9851, ip-172-31-14-61.us-west-2.compute.internal): ExecutorLostFailure (executor 14 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
1
最佳答案
您只需在spark-submit命令中提供配置即可。例如：
spark-submit --master yarn-client --conf spark.yarn.executor.memoryOverhead=4096 --num-executors 10 --executor-memory 8G --executor-cores 6 ...
关于python - 如何从AWSCLI for EMR设置YARN memoryOverhead，我们在Stack Overflow上找到一个类似的问题：https://stackoverflow.com/questions/37286429/