问题描述
我正在运行spark批处理作业,并使用SparkSession
,因为我需要很多spark-sql功能来处理我的每个组件.SparkContext
在我的父组件中初始化,并作为SparkSession
传递给子组件.
I'm running a spark batch job and uses SparkSession
as I need a lot of spark-sql features to process in each of my components. The SparkContext
is initialized in my parent component and been passed to the child components as SparkSession
.
在我的一个子组件中,我想向我的SparkContext
添加另外两个配置.因此,我需要从SparkSession
中检索SparkContext
,停止它,并使用其他配置重新创建SparkSession
.为此,如何从SparkSession检索SparkContext ?
In one of my child components, I wanted to add two more configurations to my SparkContext
. Hence, I need to retrieve the SparkContext
from the SparkSession
, stop it and recreate the SparkSession
with the additional configuration. To do so, how can I retrieve SparkContext from SparkSession?
推荐答案
只需作为答案发布-可以使用spark.sparkContext
(无括号)从SparkSession访问SparkContext
Just to post as an answer - the SparkContext can be accessed from SparkSession using spark.sparkContext
(no parenthesis)
这篇关于从SparkSession检索SparkContext的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!