如何在不使用GUI的情况下创建，更新和删除气流变量？

本文介绍了如何在不使用GUI的情况下创建，更新和删除气流变量？的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！

问题描述

我一直在学习气流并为ETL管道编写DAG。它涉及使用AWS环境（S3，Redshift）。在将数据存储到redshift之后，它负责将数据从一个存储桶复制到另一个存储桶。我将存储桶名称和前缀存储为气流中的变量，您必须为其打开GUI并手动添加它们。

I have been learning airflow and writing DAGs for an ETL pipeline. It involves using the AWS environment (S3, Redshift). It deals with copying data from one bucket to another after storing it in redshift. I am storing bucket names and prefixes as Variables in airflow for which you have to open the GUI and add them manually.

这是最安全，使用最广泛的做法行业以下选项中的

Which is the most safest and widely used practice in the industry out of the following options

我们可以使用 airflow.cfg 来存储变量（）并在我们的DAG中访问它们？

使用自定义配置文件并使用 configparser

 
 使用GUI添加变量

Can we use airflow.cfg to store our variables (bucket names) and access them in our DAGs?
Use a custom configuration file and parse its contents using configparser
Use the GUI to add variables

`推荐答案`

摘要：您可以使用airflow cli来执行json文件中变量的导入操作。您可以使用以下命令 airflow variables -i [1]并通过airflow CICD管道进行构建或手动运行它。那应该处理插入/更新的情况。对于删除，您可以显式调用 airflow变量-x ，我不认为当前可以在airflow中进行批量删除。

To summary: you can use airflow cli to perform an import operation of variables from a json file. You could use the following command airflow variables -i[1] and build it via airflow CICD pipeline or manually run it. That should handle the insert/update case. For deletion, you can call airflow variables -x explicitly, I don't think currently you can do a batch delete in airflow now.

您可以使JSON文件看起来像具有键值的以下格式：

You can have a JSON file looks like the following format with key value:

{
    "foo1": "bar1",
    "foo2": "bar2"
}

这里要注意的一件事：您可以将变量视为键值存储，因此请确保在导入时没有重复的键（否则可能会以意外的结果覆盖它）

One thing to note here: you can treat the variable as key-value storage, so make sure you don't have duplicated keys when you import (otherwise you might override it with unexpected result)

[1] airflow.apache.org/cli.html#variables

                        这篇关于如何在不使用GUI的情况下创建，更新和删除气流变量？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！

Airflow