问题描述
我正在使用Bigquery python库将csv格式的Bigquery表中的数据导出到GCS中.
I am using Bigquery python libraries to export data from Bigquery tables into GCS in csv format.
我给出了一个通配符模式，假设某些表可以超过1 GB
I have given a wildcard pattern assuming some tables can be more than 1 GB
有时，即使表只有几MB，它也会创建多个文件，有时只是创建一个文件.
Sometimes even though table is few MB it creates multiple files and sometimes just it creates just 1 file.
这背后有逻辑吗?
我的导出工作流程如下:
My export workflow is the following:
project = bq_project dataset_id = bq_dataset_id table_id = bq_table_id
bucket_name =bq_bucket_name workflow_name=workflow_nm
csv_file_nm=workflow_nm+"/"+csv_file_prefix_in_gcs+'*'client =
bigquery.Client() destination_uri = "gs://{}/{}".format(bucket_name,
csv_file_nm) dataset_ref = client.dataset(dataset_id, project=project)
table_ref = dataset_ref.table(table_id) destination_table =
client.get_table(dataset_ref.table(table_id)) configuration =
bigquery.job.ExtractJobConfig() configuration.destination_format='CSV' –
csv_file_nm=workflow_nm+"/"+csv_file_prefix_in_gcs
推荐答案
我认为这是导出的预期行为. Bigquery Export 文档指定以下内容:
I think this is an intended behaviour of the export. The Bigquery Export documentation specifies the following:
这与您在导出中看到的行为相对应.
This corresponds to the behavior you are seeing in your exports.
这篇关于Google bigquery导出表到Google Cloud存储中的多个文件，有时一个文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！