且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

Spark MLlib示例,NoSuchMethodError:org.apache.spark.sql.SQLContext.createDataFrame()

更新时间:2023-11-18 14:58:28

对于我来说,现在正在工作,仅作记录,请引用@MartinSenne答案.

It's working now for me, and just for the record, referencing @MartinSenne answer.

我所做的如下:

  1. 清除文件夹项目"下的所有编译文件
  2. scala版本 2.10.4 (以前使用2.11.4)
  3. 将spark-sql更改为:" org.apache.spark" %%"spark-sql"%"1.4.1"%提供"
  4. 将MLlib更改为:"org.apache.spark" %%"spark-mllib"%"1.4.1"%提供"
  1. clear all compile files under folder "project"
  2. scala version 2.10.4 (previously using 2.11.4)
  3. change spark-sql to be: "org.apache.spark" %% "spark-sql" % "1.4.1" % "provided"
  4. change MLlib to be: "org.apache.spark" %% "spark-mllib" % "1.4.1" % "provided"

@note:

  1. 我已经启动了一个Spark集群,并使用" sh spark-submit/path_to_folder/hello/target/scala-2.10/hello_2.10-1.0.jar "将jar提交给Spark掌握.如果使用sbt通过命令" sbt run "运行将失败.
  2. 当从scala-2.11更改为scala-2.10时,请记住jar文件路径和名称也将从" scala-2.11/hello_2.11-1.0.jar "改为" scala-2.10/hello_2.10-1.0.jar ".当我重新打包所有内容时,我忘记更改jar名称的Submit job命令,因此我将其打包为"hello_2.10-1.0.jar",但提交了"hello_2.10-1.0.jar",这给我带来了额外的问题. ..
  3. 我尝试了"val sqlContext = 新的org.apache.spark.sql.SQLContext(sc)"和"val sqlContext = 新的org.apache.spark.sql.hive". HiveContext(sc)",两者均与方法 createDataFrame()
  4. 一起使用
  1. I've already started a Spark cluster and I use "sh spark-submit /path_to_folder/hello/target/scala-2.10/hello_2.10-1.0.jar" to submit jar to Spark master. If use sbt to run by command "sbt run" will fail.
  2. when changing from scala-2.11 to scala-2.10, remember that the jar file path and name will also change from "scala-2.11/hello_2.11-1.0.jar" to "scala-2.10/hello_2.10-1.0.jar". when I re-packaged everything, I forgot to change the submit job command for the jar name, so I package into "hello_2.10-1.0.jar" but submitting "hello_2.10-1.0.jar" which caused me extra problem...
  3. I tried both "val sqlContext = new org.apache.spark.sql.SQLContext(sc)" and "val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)", both are working with method createDataFrame()