在Datalab中查询Hive表时出现问题

问题描述

我已经创建了一个具有更新的init操作的dataproc集群，以安装datalab.

I have create a dataproc cluster with an updated init action to install datalab.

一切正常，除了当我从Datalab笔记本查询Hive表时，我遇到了

All works fine, except that when I query a Hive table from the Datalab notebook, i run into

hc.sql("""select * from invoices limit 10""")

"java.lang.ClassNotFoundException: Class com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem not found" exception

创建集群

gcloud beta dataproc clusters create ds-cluster \
--project my-exercise-project \
--region us-west1 \
--zone us-west1-b \
--bucket dataproc-datalab \
--scopes cloud-platform  \
--num-workers 2  \
--enable-component-gateway  \
--initialization-actions gs://dataproc_mybucket/datalab-updated.sh,gs://dataproc-initialization-actions/connectors/connectors.sh  \
--metadata 'CONDA_PACKAGES="python==3.5"'  \
--metadata gcs-connector-version=1.9.11

datalab-updated.sh

  -v "${DATALAB_DIR}:/content/datalab" ${VOLUME_FLAGS} datalab-pyspark; then
    mkdir -p ${HOME}/datalab
    gcloud source repos clone datalab-notebooks ${HOME}/datalab/notebooks

在datalab笔记本中

from pyspark.sql import HiveContext
hc=HiveContext(sc)
hc.sql("""show tables in default""").show()
hc.sql("""CREATE EXTERNAL TABLE IF NOT EXISTS INVOICES
      (SubmissionDate DATE, TransactionAmount DOUBLE, TransactionType STRING)
      STORED AS PARQUET
      LOCATION 'gs://my-exercise-project-ds-team/datasets/invoices’""")
hc.sql("""select * from invoices limit 10""")

更新

spark._jsc.hadoopConfiguration().set('fs.gs.impl', 'com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem')
spark._jsc.hadoopConfiguration().set('fs.gs.auth.service.account.enable', 'true')
spark._jsc.hadoopConfiguration().set('google.cloud.auth.service.account.json.keyfile', "~/Downloads/my-exercise-project-f47054fc6fd8.json")

更新2(datalab-updated.sh)

function run_datalab(){
  if docker run -d --restart always --net=host  \
      -v "${DATALAB_DIR}:/content/datalab" ${VOLUME_FLAGS} datalab-pyspark; then
    mkdir -p ${HOME}/datalab
    gcloud source repos clone datalab-notebooks ${HOME}/datalab/notebooks
    echo 'Cloud Datalab Jupyter server successfully deployed.'
  else
    err 'Failed to run Cloud Datalab'
  fi
}

推荐答案

您应使用 Datalab初始化操作，用于在Dataproc集群上安装Datalab:

You should use Datalab initialization action to install Datalab on Dataproc cluster:

gcloud dataproc clusters create ${CLUSTER} \
    --image-version=1.3 \
    --scopes cloud-platform \
    --initialization-actions=gs://dataproc-initialization-actions/datalab/datalab.sh

此配置单元在Datalab中与GCS一起使用后即可使用

After this Hive works with GCS out of the box in Datalab:

from pyspark.sql import HiveContext
hc=HiveContext(sc)
hc.sql("""SHOW TABLES IN default""").show()

输出:

+--------+---------+-----------+
|database|tableName|isTemporary|
+--------+---------+-----------+
+--------+---------+-----------+

在Datalab中使用Hive在GCS上创建外部表:

Creating external table on GCS using Hive in Datalab:

hc.sql("""CREATE EXTERNAL TABLE IF NOT EXISTS INVOICES
      (SubmissionDate DATE, TransactionAmount DOUBLE, TransactionType STRING)
      STORED AS PARQUET
      LOCATION 'gs://<BUCKET>/datasets/invoices'""")

输出:

DataFrame[]

在Datalab中使用Hive查询GCS表:

Querying GCS table using Hive in Datalab:

hc.sql("""SELECT * FROM invoices LIMIT 10""")

输出:

DataFrame[SubmissionDate: date, TransactionAmount: double, TransactionType: string]

这篇关于在Datalab中查询Hive表时出现问题的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！