Apache Beam - 跳过管道步骤

问题描述

我正在使用 Apache Beam 设置一个由 2 个主要步骤组成的管道:

I'm using Apache Beam to set up a pipeline consisting of 2 main steps:

• 使用光束变换来变换数据
• 将转换后的数据加载到 BigQuery

管道设置如下所示:

myPCollection = (org.apache.beam.sdk.values.PCollection<myCollectionObjectType>)myInputPCollection
                .apply("do a parallel transform"),
                     ParDo.of(new MyTransformClassName.MyTransformFn()));

 myPCollection
    .apply("Load BigQuery data for PCollection",
            BigQueryIO.<myCollectionObjectType>write()
            .to(new MyDataLoadClass.MyFactTableDestination(myDestination))
            .withFormatFunction(new MyDataLoadClass.MySerializationFn())

我看过这个问题:

Apache Beam:在已构建的管道中跳过步骤

这表明在步骤 1 中的并行转换之后，我可能能够以某种方式动态更改我可以将数据传递到的输出.

which suggests that I may be able to somehow dynamically change which output I can pass data to, following the parallel transform in step 1.

我该怎么做?我不知道如何选择是否将 myPCollection 从第 1 步传递到第 2 步.如果第 1 步的 myPCollection 中的对象是，我需要跳过第 2 步null.

How do I do this? I don't know how to choose whether or not to pass myPCollection from step 1 to step 2. I need to skip step 2 if the object in myPCollection from step 1 is null.

推荐答案

当您在下一步中不想要它时，不要从 MyTransformClassName.MyTransformFn 发出元素，因为像这样的例子:

You just don't emit the element from your MyTransformClassName.MyTransformFn when you don't want it in the next step, for example something like this:

class MyTransformClassName.MyTransformFn extends...
  @ProcessElement
  public void processElement(ProcessContext c, ...) {
    ...
    result = ...
    if (result != null) {
       c.output(result);   //only output something that's not null
    }
  }

这样 null 就不会到达下一步.

This way nulls don't reach the next step.

有关更多详细信息，请参阅指南的 ParDo 部分:https://beam.apache.org/documentation/programming-guide/#pardo

See the ParDo section of the guide for more details: https://beam.apache.org/documentation/programming-guide/#pardo

这篇关于Apache Beam - 跳过管道步骤的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！