如何在Beam 2.0中的复合PTransform中获得PipelineOptions?

问题描述

升级到Beam 2.0之后，Pipeline类不再具有getOptions()类.我有一个复合PTransform，它依赖于获取其expand方法中的选项:

After upgrading to Beam 2.0 the Pipeline class doesn't have getOptions() class anymore.I have a composite PTransform that relies on getting the options in the its expand method:

public class MyCompositeTransform extends PTransform<PBegin, PDone> {
    @Override
    public PDone expand(PBegin input) {
        Pipeline pipeline = input.getPipeline();
        MyPipelineOptions options = pipeline.getOptions().as(MyPipelineOptions.class);
        ...
    }
}

在Beam 2.0中，似乎根本没有办法使用expand方法访问PipelineOptions.

In Beam 2.0 there doesn't seem to be a way to access the PipelineOptions at all in the expand method.

有什么选择?

推荐答案

Pablo的答案正确.我还要澄清一下，PipelineOptions的管理方式发生了重大变化.

Pablo's answer is right on. I want to also clarify that there is a major change in how PipelineOptions are managed.

您可以使用它们来解析参数并将其传递给main程序(或构建管道的任何代码)，但是从技术上讲，它们与配置管道运行方式的PipelineOptions独立.

You can use them to parse and pass around the arguments to your main program (or whatever code builds your pipeline) but these are technically independent from the PipelineOptions that configure how your pipeline is run.

在Beam中，Pipeline已完全构建，只有在此之后，您才选择PipelineRunner和PipelineOptions来控制管道的运行方式.管道本身实际上没有选项.

In Beam, the Pipeline is fully constructed, and only afterwards you choose a PipelineRunner and PipelineOptions to control how the pipeline is run. The pipeline itself does not actually have options.

如果您确实希望PTransform的行为(而不是其扩展名)使用动态获取的某些选项，则应使您的PTransform接受ValueProvider 像这样的例子WriteFiles ，您可以定义返回ValueProvider 如ValueProviderTest

If you do want the behavior of your PTransform (not its expansion) to use some option that is obtained dynamically, you should make your PTransform accept a ValueProvider like this example in WriteFiles and you can define a pipeline option that returns a ValueProvider like here in ValueProviderTest

这篇关于如何在Beam 2.0中的复合PTransform中获得PipelineOptions?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！