且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

气流:在下一个任务中获取上一个任务 ID

更新时间:2021-08-12 06:48:36

我之前没有在我的任何 DAG 中使用过 Jinja 模板,但是我遇到了类似的问题,我需要从特定的对象中检索 XCOM 值具有动态生成的 task_id 的任务.

I haven't used Jinja templating in any of my DAGs before, but I have been faced with similar problems where I was needing to retrieve XCOM values from a particular task that has a dynamically generated task_id.

您可以按照在 T2 中定义 task_id 的相同方式在 T3 中定义 task_ids.例如:

You could define the task_ids in T3 in the same way you defined the task_id in T2. For example:

source_list = ['account', 'sales']

for source_type in source_list:

    task_id='compute_next_gather_time_for_' + source_type

    t2 = PythonOperator(
                task_id=task_id,
                python_callable=compute_next_gather_time,
                provide_context=True,
                trigger_rule=TriggerRule.ALL_SUCCESS,
                op_args=[source_type],
                retries=3
            )

    t3 = SimpleHttpOperator(
                task_id='request_' + source_type + '_report',
                method='POST',
                http_conn_id='abc',
                endpoint=endpoint,
                data=json.dumps({
                    "query": {
                        "start": "{{ task_instance.xcom_pull(task_ids=task_id) }}",
                        "stop": str(yesterday),
                        "fields": [
                            1
                        ]
                    }
                }),
                headers={"Content-Type": "application/json", "Authorization": 'abc'},
                response_check=lambda response: True if len(response.json()) == 0 else False,
                log_response=True,
                retries=3
            )