本文介绍了如何使用Spark/Scala +代码+配置通过远程Hive存储(S3 ORC)在Sparksql中加载文件的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
intellij(spark)---> Hive(Remote)---存储在S3上(orc格式)无法通过spark/scala读取远程Hive表.
intellij(spark)--->Hive(Remote)---storage on S3(orc format)Not able to read remote Hive table through spark/scala.
能够读取表模式,但无法读取表.
was able to read table schema but not able to read table.
import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import org.apache.spark.sql.{Encoders, SparkSession}
import org.apache.spark.sql.hive.HiveContext
import org.apache.spark.sql.hive.orc._
import org.apache.spark.sql.types.StructType
object mainclas {
def main(args: Array[String]): Unit = {
val spark = SparkSession.builder
.master("local[*]")
.appName("hivetable")
.config("hive.metastore.uris", "thrift://10.20.30.40:9083")
.config("access-key","PQHFFDEGGDDVDVV")
.config("secret-key","FFGSGHhjhhhdjhJHJHHJGJHGjHH")
.config("format", "orc")
.enableHiveSupport()
.getOrCreate()
val res = spark.sqlContext.sql("show tables").show()
val res1 =spark.sql("select *from ace.visit limit 5").show()
}
}`
推荐答案
尝试一下:
val spark = SparkSession.builder
.master("local[*]")
.appName("hivetable")
.config("hive.metastore.uris", "thrift://10.20.30.40:9083")
.config("fs.s3n.awsAccessKeyId","PQHFFDEGGDDVDVV")
.config("fs.s3n.awsSecretAccessKey","FFGSGHhjhhhdjhJHJHHJGJHGjHH")
.config("format", "orc")
.enableHiveSupport()
.getOrCreate()
这篇关于如何使用Spark/Scala +代码+配置通过远程Hive存储(S3 ORC)在Sparksql中加载文件的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!