airflow

Airflow Generate Dynamic Tasks in Single DAG , Task N+1 is Dependent on TaskN

送分小仙女□ 提交于 2020-07-05 03:27:07
问题 When generating tasks dynamically, I need to have Task 2 be dependent of Task 1, Task1 >> Task 2 or task2.set_upstream(task1). Since the task_ids are evaluated, or seem to be upfront, I cannot set the dependency in advance, any help would be appreciated. The Component(I) tasks generate fine,

Airflow Generate Dynamic Tasks in Single DAG , Task N+1 is Dependent on TaskN

☆樱花仙子☆ 提交于 2020-07-05 03:26:06
问题 When generating tasks dynamically, I need to have Task 2 be dependent of Task 1, Task1 >> Task 2 or task2.set_upstream(task1). Since the task_ids are evaluated, or seem to be upfront, I cannot set the dependency in advance, any help would be appreciated. The Component(I) tasks generate fine,

Airflow : dag run with execution_date = trigger_date = fixed_schedule

房东的猫 提交于 2020-07-04 13:10:49
问题 in airflow, I would like to run a dag each monday at 8am (the execution_date should be of course "current day monday 8 am"). The relevant parameters to set up for this workflow are : start_date : "2018-03-19" schedule_interval : "0 8 * * MON" I expect to see a dag run every monday at 8am . The

Schedule a DAG in airflow to run for every 5 minutes , starting from today i.e., 2019-12-18

那年仲夏 提交于 2020-06-29 06:00:55
问题 I am trying to run a DAG for every 5 minutes starting from today(2019-12-18). I defined my start date as start_date:dt.datetime(2019, 12, 18, 10, 00, 00) and schedule interval as schedule_interval= '*/5 * * * *' . When I start the airflow scheduler I don't see any of my tasks running. But when I

Helm stable/airflow - Custom values for Airflow deployment with Shared Persistent Volume using Helm chart failing

拜拜、爱过 提交于 2020-06-28 03:28:42
问题 Objective I want to deploy Airflow on Kubernetes where pods have access to the same DAGs, in a Shared Persistent Volume. According to the documentation (https://github.com/helm/charts/tree/master/stable/airflow#using-one-volume-for-both-logs-and-dags), it seems I have to set and pass these values

How to use xcom_push=True and auto_remove=True at the same time when using DockerOperator?

廉价感情. 提交于 2020-06-27 16:49:34
问题 Problem When running DockerOperator with xcom_push=True , xcom_all=True and auto_remove=True , the task raises an error as if the container is deleted before reading its STDOUT . Example Consider the following DAG as an example: from datetime import datetime, timedelta from airflow import DAG

Google Cloud Composer(Airflow) - dataflow job inside a DAG executes successfully, but the DAG fails

早过忘川 提交于 2020-06-27 07:29:27
问题 My DAG looks like this default_args = { 'start_date': airflow.utils.dates.days_ago(0), 'retries': 0, 'dataflow_default_options': { 'project': 'test', 'tempLocation': 'gs://test/dataflow/pipelines/temp/', 'stagingLocation': 'gs://test/dataflow/pipelines/staging/', 'autoscalingAlgorithm': 'BASIC',

Airflow - Skip future task instance without making changes to dag file

蓝咒 提交于 2020-06-26 06:17:06
问题 I have a DAG 'abc' scheduled to run every day at 7 AM CST and there is task 'xyz' in that DAG. For some reason, I do not want to run one of the tasks 'xyz' for tomorrow's instance. How can I skip that particular task instance? I do not want to make any changes to code as I do not have access to