在EMR群集上运行的Spark作业. system.exit(0)用于正常完成工作，但仍然无法执行EMR

问题描述

在火花作业中.我正在使用if file not found system.exit(0).它应该可以正常完成工作.在本地已成功完成.但是当我运行EMR时.步骤失败.

解决方案

EMR使用 YARN 用于集群管理和启动Spark应用程序.因此，当您在EMR中使用--deploy-mode: cluster运行Spark应用程序时，Spark应用程序代码不会单独在JVM中运行，而是由 ApplicationMaster 类. >

浏览ApplicationMaster代码可以解释尝试执行System.exit()时发生的情况.用户应用程序在 startUserApplication ，然后在用户应用程序返回后调用finish方法.但是，当您调用System.exit(0)时，执行的是关闭钩子，它会看到您的代码未成功完成，并将其标记为EXIT_EARLY失败.它也有这个有用的注释:

  // The default state of ApplicationMaster is failed if it is invoked by shut down hook.
  // This behavior is different compared to 1.x version.
  // If user application is exited ahead of time by calling System.exit(N), here mark
  // this application as failed with EXIT_EARLY. For a good shutdown, user shouldn't call
  // System.exit(0) to terminate the application.

In spark job. I am using if file not found the system.exit(0). It should gracefully complete the job. Locally It is successfully completed. But when I am running on EMR. Step is failing.

解决方案

EMR uses YARN for cluster management and launching Spark applications. So when you're running a Spark app with --deploy-mode: cluster in EMR, the Spark application code is not running in a JVM on its own, but is rather executed by the ApplicationMaster class.

Browsing through the ApplicationMaster code can explain what happens when you try to execute System.exit(). The user application is launched in startUserApplication, and then the finish method is called after the user application returns. However, when you call System.exit(0), what is executed instead is a shutdown hook which sees that your code hasn't finished successfully, and marks it as an EXIT_EARLY failure. It also has this useful comment:

  // The default state of ApplicationMaster is failed if it is invoked by shut down hook.
  // This behavior is different compared to 1.x version.
  // If user application is exited ahead of time by calling System.exit(N), here mark
  // this application as failed with EXIT_EARLY. For a good shutdown, user shouldn't call
  // System.exit(0) to terminate the application.

这篇关于在EMR群集上运行的Spark作业. system.exit(0)用于正常完成工作，但仍然无法执行EMR的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！