且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

无法使用mongo-hadoop连接器将Hive与MongoDB连接

更新时间:2022-10-31 15:49:33

您需要明确注册jar。在Hive脚本中,使用 ADD JAR 命令来包含这些JAR(核心,配置单元和Java驱动程序),例如 ADD JAR / path- to / mongo-hadoop-hive-< version> .jar;



如果您是从Hive shell运行,请像这样使用。

hive> ADD JAR /path-to/mongo-hadoop-hive-<version>.jar;



然后执行您的查询。


I am trying to install and configure hive with mongo-hadoop-core 2.0.2, for the first time. I have installed hadoop 2.8.0, Hive 2.1.1 and MongoDB 3.4.6. and everything works fine when running individually.

My problem is, I am not able to connect MongoDB with Hive. I am using mongo-Hadoop connector for this as mentioned here https://github.com/mongodb/mongo-hadoop/wiki/Hive-Usage The required jars are added to Hadoop and Hive lib. Even I add them in hive.sh or runtime from hive console. I am getting error while executing Create table query

My Query is

CREATE EXTERNAL TABLE testHive
(
    id STRING,
    name STRING
)
STORED BY 'com.mongodb.hadoop.hive.MongoStorageHandler'
WITH SERDEPROPERTIES('mongo.columns.mapping'='{"id":"_id","name":"name"}')
TBLPROPERTIES('mongo.uri'='mongodb://localhost:27017/hiveDb.testHive');

And I get the following error

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. com/mongodb/hadoop/io/BSONWritable
hive> ERROR hive.ql.exec.DDLTask - java.lang.NoClassDefFoundError: com/mongodb/hadoop/io/BSONWritable
    at com.mongodb.hadoop.hive.BSONSerDe.initialize(BSONSerDe.java:132)
    at org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:537)
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:424)
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:411)
    at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:279)
    at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:261)

It shows that com/mongodb/hadoop/io/BSONWritable class is not in classpath but I have added the required(mongo-hadoop-core.jar) jar and class are present in the jar. The version of jars I am using

mongo-hadoop-core 2.0.2,

mongo-hadoop-hive 2.0.2,

mongo-java-driver 3.0.2

Thanks

You need to register jars explicitly. In your Hive script, use ADD JAR commands to include these JARs (core, hive, and the Java driver), e.g., ADD JAR /path-to/mongo-hadoop-hive-<version>.jar;.

If you are running from Hive shell, use like this.

hive> ADD JAR /path-to/mongo-hadoop-hive-<version>.jar;

Then execute your query.