且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

基于 Docker 的解决方案中的 Apache Spark 和 Apache Airflow 连接

更新时间:2023-11-28 18:59:40

如果您使用 SparkSubmitOperator,则与 master 的连接将设置为Yarn";默认情况下,无论您在 Python 代码中设置了哪个 master,您都可以通过在其构造函数中指定 conn_id 来覆盖 master,条件是您已经在Admin->Connection"处创建了上述 conn_idAirflow Web 界面中的菜单.希望能帮到你.

If you use SparkSubmitOperator the connection to master will be set as "Yarn" by default regardless master which you set on your python code, however, you can override master by specifying conn_id through its constructor with the condition you have already created the aforementioned conn_id at "Admin->Connection" menu in Airflow Web Interface. I hope it can help you.