Apache Beam和BigQuery

问题描述

我正在尝试执行apache beam sdk 2.4和库com.google.cloud.bigquery但出现异常

I'm trying to execute apache beam sdk 2.4 and the libraries com.google.cloud.bigquery but thow exception

Exception in thread main java.lang.NoSuchMethodError com.google.api.client.googleapis.services.json.AbstractGoogleJsonClient$Builder.setBatchPath(LjavalangString;)LcomgoogleapiclientgoogleapisservicesAbstractGoogleClient$Builder;
at com.google.api.services.bigquery.Bigquery$Builder.setBatchPath(Bigquery.java3519)

import com.google.cloud.bigquery.*;

<dependency>
    <groupId>com.google.cloud.dataflow</groupId>
    <artifactId>google-cloud-dataflow-java-sdk-all</artifactId>
    <version>[2.4.0, 2.99)</version>
</dependency>

<!-- slf4j API frontend binding with JUL backend -->
<dependency>
    <groupId>org.slf4j</groupId>
    <artifactId>slf4j-api</artifactId>
    <versi

on>1.7.14</version>
    </dependency>

   <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-jdk14</artifactId>
        <version>1.7.14</version>
    </dependency>

    <dependency>
        <groupId>com.google.cloud</groupId>
        <artifactId>google-cloud-bigquery</artifactId>
        <version>1.35.0</version>
    </dependency>

</dependencies>

the code example apache beam 2.4 using

import com.google.api.services.bigquery.model.TableFieldSchema;
import com.google.api.services.bigquery.model.TableRow;
import com.google.api.services.bigquery.model.TableSchema;

内部使用什么库?

我应该使用哪个库? com.google.api.services.bigquery还是com.google.cloud.bigquery?

What library should I use? com.google.api.services.bigquery or com.google.cloud.bigquery?

推荐答案

Beam 2.4正在使用 com.google.api.services.bigquery v2-rev374-1.22.0.这也是应与Beam 2.4一起使用的库. com.google.api.services.bigquery库已过期，并且有一个打开进行升级.不幸的是，Beam中的其他组件取决于与com.google.cloud.bigquery不兼容的库，在使用新库之前，需要对其进行升级.

Beam 2.4 is using com.google.api.services.bigquery v2-rev374-1.22.0. This is also the library that you should use with Beam 2.4. The com.google.api.services.bigquery library is out of date and there is an open issue to upgrade it. Unfortunately other components in Beam depend on libraries which are incompatible with com.google.cloud.bigquery, and those need to be upgraded before the new library can be used.

如果您尚未这样做，则应查看 BigQuery IO ，它为Beam的常见BigQuery操作提供了包装.

If you aren't already doing so, you should take a look at the BigQuery IO, which provides a wrapper to common BigQuery operations for Beam.

这篇关于Apache Beam和BigQuery的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！