且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

如何在SparkSQL中以编程方式连接到Hive Metastore?

更新时间:2023-11-30 17:02:40

对于Spark 1.x,您可以设置:

  System.setProperty(hive.metastore.uris,thrift:// METASTORE:9083); 

final SparkConf conf = new SparkConf();
SparkContext sc = new SparkContext(conf);
HiveContext hiveContext = new HiveContext(sc);



  final SparkConf conf = new SparkConf(); 
SparkContext sc = new SparkContext(conf);
HiveContext hiveContext = new HiveContext(sc);
hiveContext.setConf(hive.metastore.uris,thrift:// METASTORE:9083);

更新如果您的Hive已被Kerberized

尝试在创建HiveContext之前设置它们:

  System.setProperty(hive.metastore.sasl .enabled,true); 
System.setProperty(hive.security.authorization.enabled,false);
System.setProperty(hive.metastore.kerberos.principal,hivePrincipal);
System.setProperty(hive.metastore.execute.setugi,true);


I'm using HiveContext with SparkSQL and I'm trying to connect to a remote Hive metastore, the only way to set the hive metastore is through including the hive-site.xml on the classpath (or copying it to /etc/spark/conf/).

Is there a way to set this parameter programmatically in a java code without including the hive-site.xml ? If so what is the Spark configuration to use ?

For Spark 1.x, you can set with :

System.setProperty("hive.metastore.uris", "thrift://METASTORE:9083");

final SparkConf conf = new SparkConf();
SparkContext sc = new SparkContext(conf);
HiveContext hiveContext = new HiveContext(sc);

Or

final SparkConf conf = new SparkConf();
SparkContext sc = new SparkContext(conf);
HiveContext hiveContext = new HiveContext(sc);
hiveContext.setConf("hive.metastore.uris", "thrift://METASTORE:9083");

Update If your Hive is Kerberized :

Try setting these before creating the HiveContext :

System.setProperty("hive.metastore.sasl.enabled", "true");
System.setProperty("hive.security.authorization.enabled", "false");
System.setProperty("hive.metastore.kerberos.principal", hivePrincipal);
System.setProperty("hive.metastore.execute.setugi", "true");