本文介绍了整个DAG的气流取决于_过去的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

是否有一种方法可以对整个DagRun使用 depends_on_past ,而不仅仅是将其应用于Task?

Is there a way in airflow of using the depends_on_past for an entire DagRun, not just applied to a Task?

我每天都有DAG,星期五DagRun在第四个任务上出错,但是星期六和星期日DagRun仍按计划运行。使用 depends_on_past = True 可以使DagRun在相同的第4个任务上暂停,但是前三个任务仍然可以运行。

I have a daily DAG, and the Friday DagRun errored on the 4th task however the Saturday and Sunday DagRuns still ran as scheduled. Using depends_on_past = True would have paused the DagRun on the same 4th task, however the first 3 tasks would still have run.

我可以在DagRun数据库表中看到一个 state 列,其中包含星期五DagRun的失败。我想要的是一种配置DagRun的方法,以使其在先前的DagRun失败时不启动,直到发现以前失败的Task才启动并运行。

I can see in the DagRun DB table there is a state column that contains failed for the Friday DagRun. What I want is a way configuring a DagRun to not start if the previous DagRun failed, not start and run until finding a Task that previously failed.

有人知道这是否可能吗?

Does anyone know if this is possible?

推荐答案

在您的身边第一个任务,设置 depends_on_past = True wait_for_downstream = True ,组合将导致仅运行当前的dag运行如果最后一次运行成功。

At your first task, set depends_on_past=True and wait_for_downstream=True, the combination will result in that current dag-run runs only if the last run succeeded.

因为在当前dag-run设置第一个任务会等待之前的
(depends_on_past)和所有任务(wait_for_downstream)成功

Because by setting the first task at current dag-run would waits for previous (depends_on_past) and all tasks (wait_for_downstream) to succeed

这篇关于整个DAG的气流取决于_过去的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!

07-04 06:58