问题描述
由于某种原因,Airflow似乎不会触发具有每周计划间隔的dag的最新运行。
For some reason, Airflow doesn't seem to trigger the latest run for a dag with a weekly schedule interval.
当前日期:
$ date
$ Tue Aug 9 17:09:55 UTC 2016
DAG:
from datetime import datetime
from datetime import timedelta
from airflow import DAG
from airflow.operators.bash_operator import BashOperator
dag = DAG(
dag_id='superdag',
start_date=datetime(2016, 7, 18),
schedule_interval=timedelta(days=7),
default_args={
'owner': 'Jon Doe',
'depends_on_past': False
}
)
BashOperator(
task_id='print_date',
bash_command='date',
dag=dag
)
运行调度程序
$ airflow scheduler -d superdag
您需要预计总共有四个DAG运行作为计划er应该回填7 / 18、7 / 25、8 / 1和8/8。
但是,上次运行未安排。
You'd expect a total of four DAG Runs as the scheduler should backfill for 7/18, 7/25, 8/1, and 8/8.However, the last run is not scheduled.
编辑1:
I理解Vineet,尽管这似乎并不能解释我的问题。
I understand that Vineet although that doesn’t seem to explain my issue.
在上面的示例中,DAG的开始日期是7月18日。
In my example above, the DAG’s start date is July 18.
- 第一次DAG运行:7月18日
- 第二次DAG运行:7月25日
- 第三次DAG运行:8月1日
- 第四次DAG运行:8月8日(未运行)
- First DAG Run: July 18
- Second DAG Run: July 25
- Third DAG Run: Aug 1
- Fourth DAG Run: Aug 8 (not run)
其中每个DAG运行处理前一周的数据。
Where each DAG Run processes data from the previous week.
今天是8月9日,我希望第四次DAG运行已执行,执行日期为8月8日,该运行日期用于处理最后一周(八月1,直到8月8日)。
Today being Aug 9, I would expect the Fourth DAG Run to have executed with a execution date of Aug 8 which processes data for the last week (Aug 1 until Aug 8) but it doesn’t.
推荐答案
气流始终排在上一个时段。因此,如果您计划在8月9日每天运行一次dag,则它将在8月8日将执行日期定为执行日期。同样,如果计划时间间隔是每周一次,则在8月9日,它将计划在1周后(即8月2日)进行计划,尽管该时间间隔是在8月9日执行的。这只是气流簿记。您可以在气流Wiki():
Airflow always schedules for the previous period. So if you have a dag that is scheduled to run daily, on Aug 9th, it will schedule a run with execution_date Aug 8th. Similarly if the schedule interval is weekly, then on Aug 9th, it will schedule for 1 week back i.e. Aug 2nd, though this gets run on Aug 9th itself. This is just airflow bookkeeping. You can find this in the airflow wiki (https://cwiki.apache.org/confluence/display/AIRFLOW/Common+Pitfalls):
这篇关于气流不会回填最新运行的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!