问题描述
我在VM,Cloudera计算机上使用Spark 1.6.0。
我正在尝试从Spark外壳向Hive表中输入一些数据。
为此,我正在尝试使用SparkSession。但是下面的导入无效。
scala> import org.apache.spark.sql.SparkSession
< console>:33:错误:对象SparkSession不是包org.apache.spark.sql的成员
import org.apache.spark.sql .SparkSession
否则,我将无法执行以下语句:
val spark = SparkSession.builder.master( local [2])。enableHiveSupport()。config( hive.exec.dynamic.partition, true ).config( hive.exec.dynamic.partition.mode, nonstrict)。config( spark.sql.warehouse.dir,WarehouseLocation).config( hive.metastore.warehouse.dir, / user / hive / warehouse)。getOrCreate()
< console>:33:错误:找不到:value SparkSession
val spark = SparkSession.builder.master( local [2]) .enableHiveSupport()。config( hive.exec.dynamic.partition, true)。config( hive.exec.dynamic.partition.mode, nonstrict)。config( spark.sql.warehouse。 dir,WarehouseLocation).config( hive.metastore.warehouse.dir, / user / hive / warehouse)。getOrCreate()
有人可以告诉我我在这里犯什么错误吗?
SparkSession
,因此您应该改用 SQLContext
(或将Spark升级到最新的,最出色的)。
引用Spark 1.6.0的:
I use Spark 1.6.0 on my VM, Cloudera machine.
I'm trying to enter some data into Hive table from Spark shell.To do that, I am trying to use SparkSession. But the below import is not working.
scala> import org.apache.spark.sql.SparkSession
<console>:33: error: object SparkSession is not a member of package org.apache.spark.sql
import org.apache.spark.sql.SparkSession
And without that, I cannot execute this statement:
val spark = SparkSession.builder.master("local[2]").enableHiveSupport().config("hive.exec.dynamic.partition","true").config("hive.exec.dynamic.partition.mode", "nonstrict").config("spark.sql.warehouse.dir", warehouseLocation).config("hive.metastore.warehouse.dir","/user/hive/warehouse").getOrCreate()
<console>:33: error: not found: value SparkSession
val spark = SparkSession.builder.master("local[2]").enableHiveSupport().config("hive.exec.dynamic.partition","true").config("hive.exec.dynamic.partition.mode", "nonstrict").config("spark.sql.warehouse.dir", warehouseLocation).config("hive.metastore.warehouse.dir","/user/hive/warehouse").getOrCreate()
Can anyone tell me what mistake am I doing here ?
SparkSession
is available as of Spark 2.0 so you should be using SQLContext
instead (or upgrade your Spark to the latest and greatest 2.1.1).
Quoting Spark 1.6.0's Starting Point: SQLContext:
这篇关于为什么在Spark Shell中导入SparkSession失败,并显示“对象SparkSession不是软件包org.apache.spark.sql的成员”?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!