它现在对我有用,只是为了记录,参考 @MartinSenne 的答案.
It's working now for me, and just for the record, referencing @MartinSenne answer.
我所做的如下:
- 清除文件夹project"下的所有编译文件
- scala 版本 2.10.4(之前使用 2.11.4)
- 将 spark-sql 更改为:"org.apache.spark" %% "spark-sql" % "1.4.1" % "provided"
- 将 MLlib 更改为:"org.apache.spark" %% "spark-mllib" % "1.4.1" % "provided"
- clear all compile files under folder "project"
- scala version 2.10.4 (previously using 2.11.4)
- change spark-sql to be: "org.apache.spark" %% "spark-sql" % "1.4.1" % "provided"
- change MLlib to be: "org.apache.spark" %% "spark-mllib" % "1.4.1" % "provided"
@note:
- 我已经启动了一个 Spark 集群,我使用sh spark-submit/path_to_folder/hello/target/scala-2.10/hello_2.10-1.0.jar"将 jar 提交给 Spark掌握.如果使用 sbt 通过命令sbt run"运行将失败.
- 当从 scala-2.11 更改为 scala-2.10 时,请记住 jar 文件路径和名称 也会从scala-2.11/hello_2.11-1.0.jar"更改强>"到scala-2.10/hello_2.10-1.0.jar".当我重新打包所有东西时,我忘记更改jar名称的提交作业命令,所以我打包到hello_2.10-1.0.jar"但提交hello_2.10-1.0.jar"这给我带来了额外的问题...
- 我尝试了val sqlContext = new org.apache.spark.sql.SQLContext(sc)"和val sqlContext = new org.apache.spark.sql.hive".HiveContext(sc)",两者都使用方法 createDataFrame()
- I've already started a Spark cluster and I use "sh spark-submit /path_to_folder/hello/target/scala-2.10/hello_2.10-1.0.jar" to submit jar to Spark master. If use sbt to run by command "sbt run" will fail.
- when changing from scala-2.11 to scala-2.10, remember that the jar file path and name will also change from "scala-2.11/hello_2.11-1.0.jar" to "scala-2.10/hello_2.10-1.0.jar". when I re-packaged everything, I forgot to change the submit job command for the jar name, so I package into "hello_2.10-1.0.jar" but submitting "hello_2.10-1.0.jar" which caused me extra problem...
- I tried both "val sqlContext = new org.apache.spark.sql.SQLContext(sc)" and "val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)", both are working with method createDataFrame()