安装程序2 youtube bigquery传输（每个拥有者ID一个），使用相同的后缀，以便数据进入同一个表。该传输缺少大量数据

问题描述

通常，我每天获得的数据范围应该超过100,000条记录。但是，当我设置2 bigquery传输（每个所有者ID一个），使用相同的后缀，所以数据转到同一个表。传输中缺少大量数据。

转移状态显示成功，但有些日子我只获得4000-5000条记录，而不是100,000条记录。但有些日子我得到了100,000条记录。

问题1：

是否可能的bigquery转移只能完成部分数据？我的理解是bigquery转移是全部或没有。如果成功，那么所有的转移都完成了。如果失败，则传输零记录。

虽然我在某些情况下看到bigquery失败，但状态显示已完成。

问题2： / p>

当我建立两个传输时，两个传输都将数据传输到同一个表中，我可以看到故障率急剧增加。

在这个过程中，两次传输有可能以某种方式相互冲突吗？ 我确认了。

例如，我有帐户1和帐户2 。

1. 运行帐户1的转帐，我获得了100,000个帐户。


2. 
   

但我将帐户1的转帐和帐户2的转帐设置为同一个表格（相同的后缀） p>

1. 我运行帐户1转帐，我获得了100,000行。


2. 然后我运行帐户2转帐，丢失了刚刚由帐户1传输产生的100,000行，但我获得了5000行帐户2生成。

---

由bigquery专家提供咨询。是。我无法设置两个传输将数据转储到一个表。由于bigquery传输回填是删除并插入。

Usually, the data range I got each day should be over 100,000 records. But when I setup 2 bigquery transfer (one for each owner id), use the same suffix, so the data goes to the same table. the transfer is missing a lot of data.

The transfer status shows successful but somedays I only get a 4000-5000 records instead of 100,000+ records. But somedays I got 100,000 records.

Question 1:

Is that possible bigquery transfer only finishes transferring partial of the data? My understanding is bigquery transfer is "all or none". If it is successful, then all the transfer is done. If it fails, zero records transferred.

Although, I do see in some cases that bigquery failed but status shows complete.

Question 2:

When I set up two transfers, both transfers sending data to a same table, I can see the failure rate increases hugely.

Is it possible the two transfers somehow conflict with each other in the process?

解决方案

OK.

I confirmed.

For example, I have account 1 and account 2.

1. Running account 1's transfer, I got 100,000 rows.
2. Running account 2's transfer, I got 5,000 rows.

But I set up both account 1's transfer and account 2's transfer to the same table (the same suffix)

1. I run account 1 transfer, I got 100,000 rows.
2. Then I run account 2 transfer, I lost the the 100,000 rows that just generated by account 1 transfer, but I got 5,000 rows account 2 generated.

---

Advised by the bigquery expert. Yes. I cannot set up two transfers to dump data to one table. Due to bigquery transfer backfill is "delete and insert".

这篇关于安装程序2 youtube bigquery传输（每个拥有者ID一个），使用相同的后缀，以便数据进入同一个表。该传输缺少大量数据的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！