如何设置默认的火花日志记录级别?

问题描述

我在自己的工作站上从 pycharm 启动 pyspark 应用程序到 8 节点集群.该集群还具有编码在 spark-defaults.conf 和 spark-env.sh 中的设置

I launch pyspark applications from pycharm on my own workstation, to a 8 node cluster. This cluster also has settings encoded in spark-defaults.conf and spark-env.sh

这就是我获取 spark 上下文变量的方式.

This is how I obtain my spark context variable.

spark = SparkSession
        .builder
        .master("spark://stcpgrnlp06p.options-it.com:7087")
        .appName(__SPARK_APP_NAME__)
        .config("spark.executor.memory", "50g")
        .config("spark.eventlog.enabled", "true")
        .config("spark.eventlog.dir", r"/net/share/grid/bin/spark/UAT/SparkLogs/")
        .config("spark.cores.max", 128)
        .config("spark.sql.crossJoin.enabled", "True")
        .config("spark.executor.extraLibraryPath","/net/share/grid/bin/spark/UAT/bin/vertica-jdbc-8.0.0-0.jar")
        .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
        .config("spark.logConf", "true")
        .getOrCreate()

    sc = spark.sparkContext
    sc.setLogLevel("INFO")

我想查看我的日志中正在使用的有效配置.这一行

I want to see the effective config that is being used in my log. This line

        .config("spark.logConf", "true")

应该导致 spark api 将其有效配置作为 INFO 记录到日志中，但默认日志级别设置为 WARN，因此我看不到任何消息.

should cause the spark api to log its effective config to the log as INFO, but the default log level is set to WARN, and as such I don't see any messages.

设置这条线

sc.setLogLevel("INFO")

显示前进的 INFO 消息，但为时已晚.

shows INFO messages going forward, but its too late by then.

如何设置 spark 启动的默认日志记录级别?

How can I set the default logging level that spark starts with?

推荐答案

http://spark.apache.org/docs/latest/configuration.html#configuring-logging

配置日志记录

Spark 使用 log4j 进行日志记录.您可以通过在conf目录中添加一个log4j.properties文件进行配置.一种开始方法是复制位于那里的现有 log4j.properties.template.

Spark uses log4j for logging. You can configure it by adding a log4j.properties file in the conf directory. One way to start is to copy the existing log4j.properties.template located there.

以下关于如何登录 spark"的博客 https://www.mapr.com/blog/how-log-apache-spark 提出了一种配置 log4j 的方法，并提供了包括将 INFO 级别的日志定向到文件中的建议.

The following blog about "How to log in spark" https://www.mapr.com/blog/how-log-apache-spark suggest a way to configure log4j, and provide suggestion which includes directing INFO level logs into a file.

这篇关于如何设置默认的火花日志记录级别?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！