且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

提交作业时java.io.EOFException在星火EC2集群编程

更新时间:2023-11-15 20:07:10

浪费了大量的时间后,我发现这个问题。尽管我没有使用过的Hadoop / HDFS在我的应用程序,Hadoop的客户端的问题。问题是在Hadoop中,客户端版本,它比的Hadoop版本不同,火花是专为。星火公司的Hadoop版本1.2.1,但在我的应用程序,它为2.4。

After wasting a lot of time, I've found the problem. Despite I haven't used hadoop/hdfs in my application, hadoop client matters. The problem was in hadoop-client version, it was different than the version of hadoop, spark was built for. Spark's hadoop version 1.2.1, but in my application that was 2.4.

当我在我的应用程序改变的Hadoop客户端的版本为1.2.1,我能够执行火花code集群。

When I changed the version of hadoop client to 1.2.1 in my app, I'm able to execute spark code on cluster.