airflow-scheduler

How to delete XCOM objects once the DAG finishes its run in Airflow

北战南征 提交于 2020-01-14 08:11:12
问题 I have a huge json file in the XCOM which later I do not need once the dag execution is finished, but I still see the Xcom Object in the UI with all the data, Is there any way to delete the XCOM programmatically once the DAG run is finished. Thank you 回答1: You have to add a task depends on you

Dynamically Creating DAG based on Row available on DB Connection

和自甴很熟 提交于 2020-01-06 05:38:33
问题 I want to create a dynamically created DAG from database table query. When I'm trying to create a dynamically creating DAG from both of range of exact number or based on available object in airflow settings it's succeeded. However when I'm trying to use a PostgresHook and create a DAG for each of

Airflow DAG dynamic structure

↘锁芯ラ 提交于 2020-01-04 14:24:11
问题 I was looking for a solution where I can decide the dag structure when the dag is triggered as I'm not sure about the number of operators that I'll have to run. Please refer below for the execution sequence that I'm planning to create. |-- Task B.1 --| |-- Task C.1 --| |-- Task B.2 --| |-- Task C

airflow pass parameter from cli

牧云@^-^@ 提交于 2019-12-31 12:43:09
问题 Is there a way to pass a parameter to: airflow trigger_dag dag_name {param} ? I have a script that monitors a directory for files - when a file gets moves into the target directory I want to trigger the dag passing as a parameter the file path. 回答1: you can pass it like this: airflow trigger_dag

AirflowException: Celery command failed - The recorded hostname does not match this instance's hostname

泄露秘密 提交于 2019-12-29 08:21:51
问题 I'm running Airflow on a clustered environment running on two AWS EC2-Instances. One for master and one for the worker. The worker node though periodically throws this error when running "$airflow worker": [2018-08-09 16:15:43,553] {jobs.py:2574} WARNING - The recorded hostname ip-1.2.3.4 does