各个dataproc火花日志在哪里

各个dataproc火花日志在哪里

本文介绍了各个dataproc火花日志在哪里?的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

dataproc spark作业日志位于何处?我知道在日志记录"部分下有来自驱动程序的日志,但是执行节点呢?另外,Spark正在执行的详细步骤记录在哪里(我知道我可以在Application Master中看到它们)?我正在尝试调试似乎挂起的脚本,并且似乎冻结了火花.

Where are the dataproc spark job logs located? I know there are logs from the driver under "Logging" section but what about the execution nodes? Also, where are the detailed steps that Spark is executing logged (I know I can see them in the Application Master)? I am attempting to debug a script that seems to hang and spark seems to freeze.

推荐答案

任务日志存储在/tmp下的每个工作节点上.

The task logs are stored on each worker node under /tmp.

可以通过纱线记录汇总将它们收集在一个地方.在集群创建时设置这些属性(通过带有yarn:前缀的--properties):

It is possible to collect them in one place via yarn log aggregation. Set these properties at cluster creation time (via --properties with yarn: prefix):

  • yarn.log-aggregation-enable=true
  • yarn.nodemanager.remote-app-log-dir=gs://${LOG_BUCKET}/logs
  • yarn.log-aggregation.retain-seconds=-1
  • yarn.log-aggregation-enable=true
  • yarn.nodemanager.remote-app-log-dir=gs://${LOG_BUCKET}/logs
  • yarn.log-aggregation.retain-seconds=-1

这是一篇讨论日志管理的文章:

Here's an article that discusses log management:

https://hortonworks.com/Blog/simplifying-user-logs-management-and-access-in-yarn/

这篇关于各个dataproc火花日志在哪里?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!

09-01 20:03