且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

远程将作业提交到Spark EC2集群

更新时间:2022-03-04 06:51:13

文档位于:

http: //spark.apache.org/docs/latest/security.html#configuring-ports-for-network-security

列出所有不同的通信渠道在Spark集群中使用。如您所见,从执行器到驱动程序之间建立了连接。当您使用-deploy-mode = client 运行时,驱动程序将在笔记本电脑上运行,因此执行者将尝试与笔记本电脑建立连接。如果您的执行者在其下运行的AWS安全组阻止了您笔记本电脑的出站流量(Spark EC2脚本创建的默认安全组没有),或者您位于路由器/防火墙之后(更有可能),则它们将无法连接

lists all the different communication channels used in a Spark cluster. As you can see, there are a bunch where the connection is made from the Executor(s) to the Driver. When you run with --deploy-mode=client, the driver runs on your laptop, so the executors will try to make a connection to your laptop. If the AWS security group that your executors run under blocks outbound traffic to your laptop (which the default security group created by the Spark EC2 scripts doesn't), or you are behind a router/firewall (more likely), they fail to connect and you get the errors you are seeing.

因此,要解决此问题,必须将所有必需的端口转发到笔记本电脑,或重新配置防火墙以允许连接到端口。看到一堆端口是随机选择的,这意味着打开了广泛的端口(如果不是全部)。因此,使用群集中的-deploy-mode = cluster client 可能会减轻痛苦。

So to resolve it, you have to forward all the necessary ports to your laptop, or reconfigure your firewall to allow connection to the ports. Seeing as a bunch of the ports are chosen at random, this means opening up a wide range of, if not all ports. So probably using --deploy-mode=cluster, or client from the cluster, is less painful.