airflow-scheduler

Schedule a DAG in airflow to run for every 5 minutes , starting from today i.e., 2019-12-18

那年仲夏 提交于 2020-06-29 06:00:55
问题 I am trying to run a DAG for every 5 minutes starting from today(2019-12-18). I defined my start date as start_date:dt.datetime(2019, 12, 18, 10, 00, 00) and schedule interval as schedule_interval= '*/5 * * * *' . When I start the airflow scheduler I don't see any of my tasks running. But when I

DAG is not visible on Airflow UI

为君一笑 提交于 2020-06-17 09:34:29
问题 This is my dag file in dags folder. Code that goes along with the Airflow located at: http://airflow.readthedocs.org/en/latest/tutorial.html """ from airflow import DAG from airflow.operators.dummy_operator import DummyOperator from airflow.operators.python_operator import PythonOperator from

Airflow Scheduler is continuously issuing warning when using postgresSQL 12 as backend database

冷暖自知 提交于 2020-06-17 09:11:27
问题 While executing airflow scheduler is continue printing following messages and tasks are NOT getting picked up. [2020-02-21 09:21:20,696] {dag_processing.py:663} WARNING - DagFileProcessorManager (PID=11895) exited with exit code -11 - re-launching [2020-02-21 09:21:20,699] {dag_processing.py:556}

How to stop DAG from backfilling? catchup_by_default=False and catchup=False does not seem to work and Airflow Scheduler from backfilling

為{幸葍}努か 提交于 2020-05-23 17:49:15
问题 The setting catchup_by_default=False in airflow.cfg does not seem to work. Also adding catchup=False to the DAG doesn't work neither. Here's how to reproduce the issue. I always start from a clean slate by running airflow resetdb . As soon as I unpause the dag, the tasks start to backfill. Here's

How to Trigger a DAG on the success of a another DAG in Airflow using Python?

我的未来我决定 提交于 2020-05-14 17:47:59
问题 I have a python DAG Parent Job and DAG Child Job . The tasks in the Child Job should be triggered on the successful completion of the Parent Job tasks which are run daily. How can add external job trigger ? MY CODE from datetime import datetime, timedelta from airflow import DAG from airflow

Airflow skip current task

一曲冷凌霜 提交于 2020-01-24 12:15:08
问题 Is there a way for Airflow to skip current task from within the (Python)Operator? For example: def execute(): if condition: skip_current_task() task = PythonOperator(task_id='task', python_callable=execute, dag=some_dag) Skipping downstream tasks doesn't suit me (a solution proposed in this

Airflow skip current task

邮差的信 提交于 2020-01-24 12:15:08
问题 Is there a way for Airflow to skip current task from within the (Python)Operator? For example: def execute(): if condition: skip_current_task() task = PythonOperator(task_id='task', python_callable=execute, dag=some_dag) Skipping downstream tasks doesn't suit me (a solution proposed in this

How to skip tasks on Airflow?

余生长醉 提交于 2020-01-23 07:51:21
问题 I'm trying to understand whether Airflow supports skipping tasks in a DAG for ad-hoc executions? Lets say my DAG graph look like this: task1 > task2 > task3 > task4 And I would like to start my DAG manually from task3, what is the best way of doing that? I've read about ShortCircuitOperator , but