Google Cloud Dataproc删除BigQuery表不起作用

问题描述

我尝试使用 Dataproc中的> java客户端库，如下所示启动spark-shell:

Hi I tried to delete a table from BigQuery using java client library in Dataproc, started spark-shell as below:

spark-shell --packages com.google.cloud:google-cloud-bigquery:1.59.0

并导入以下依赖项

import com.google.cloud.bigquery.BigQuery;
import com.google.cloud.bigquery.BigQueryOptions;
import com.google.cloud.bigquery.FieldValueList;
import com.google.cloud.bigquery.Job;
import com.google.cloud.bigquery.JobId;
import com.google.cloud.bigquery.JobInfo;
import com.google.cloud.bigquery.QueryJobConfiguration;
import com.google.cloud.bigquery.QueryResponse;
import com.google.cloud.bigquery.TableResult;
import java.util.UUID;

val bigquery = BigQueryOptions.getDefaultInstance().getService()
bigquery.delete("test","temp")

这里的test和temp分别是我的数据集和表名，但是运行上面的语句后，它显示以下错误:

here test and temp are my dataset and table name respectively but after running above statement it show following error:

java.lang.NoSuchMethodError: com.google.common.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;
  at com.google.api.gax.retrying.BasicRetryingFuture.<init>(BasicRetryingFuture.java:82)
  at com.google.api.gax.retrying.DirectRetryingExecutor.createFuture(DirectRetryingExecutor.java:88)
  at com.google.api.gax.retrying.DirectRetryingExecutor.createFuture(DirectRetryingExecutor.java:74)
  at com.google.cloud.RetryHelper.run(RetryHelper.java:75)
  at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
  at com.google.cloud.bigquery.BigQueryImpl.delete(BigQueryImpl.java:386)
  at com.google.cloud.bigquery.BigQueryImpl.delete(BigQueryImpl.java:375)
  ... 48 elided

推荐答案

这是因为您在类路径上有一个较旧的没有MoreExecutors.directExecutor方法的Guava库版本(称为Hadoop/Spark依赖项).

This is because you have on classpath an older Guava library version (brought as Hadoop/Spark dependency) that does not have MoreExecutors.directExecutor method.

要解决此问题，您需要添加并阴影/重新放置(以避免与类路径上的其他库冲突)google-cloud-bigquery库及其依赖项(包括Guava)进入应用程序的.

To fix this issue you need to include and shade/relocate (to avoid conflicts with other libraries on classpath) google-cloud-bigquery library and its dependencies (including Guava) into your application's UberJar.

这是使用 Maven Shade 插件的方法的示例.

Here is an example of how to do this using Maven Shade plugin.

这篇关于Google Cloud Dataproc删除BigQuery表不起作用的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！