triggering_frequency仅可与写入BigQuery的FILE_LOADS方法一起使用

问题描述

无法为数据流流作业设置trigging_frequency.

Unable to set triggering_frequency for Dataflow Streaming job.

transformed | 'Write' >> beam.io.WriteToBigQuery(
          known_args.target_table,
          schema=schema,
          create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED,
          write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND,
          method=bigquery.WriteToBigQuery.Method.FILE_LOADS,
          triggering_frequency=5
          )

Error: triggering_frequency can only be used with FILE_LOADS method of writing to BigQuery

推荐答案

这是一个错误. WriteToBigQuery 转换有几个实现-支持流式传输中文件加载的实现是相当新的，因此在实验之后被掩盖了.

This is a bug. The WriteToBigQuery transform has a couple implementations - and the one that supports file loads in streaming is fairly new, thus it is masked behind an experiment.

因此:要激活此功能，请通过 use_beam_bq_sink 作为实验，例如:-实验use_beam_bq_sink .

So: To activate this, please pass use_beam_bq_sink as an experiment, like so: --experiments use_beam_bq_sink.

这篇关于triggering_frequency仅可与写入BigQuery的FILE_LOADS方法一起使用的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！