且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

将大型XML文件拆分为Hadoop的易管理部分

更新时间:2023-11-25 14:06:52

我认为 Cloud9项目在UMD 可能会帮助你。



图书馆提供了 XMLInputFormat class 可能有用。



Cloud9文档中的这个页面是关于如何处理***在MapReduce中的XML转储


Is there a input class to deal with [multiple] large XML files based on their tree structure in Hadoop? I have a set of XML files that are of the same schema, but I need to split them into sections of data, as opposed to breaking the sections up.

For example the XML file would be:

<root>
  <parent> data </parent>
  <parent> more data</parent>
  <parent> even more data</parent>
</root>

I would define each section as: /root/parent.

What I'm asking is: Is there a record input reader already included for Hadoop to do this?

I think the Cloud9 project at UMD might help you with this.

The library provides has an XMLInputFormat class which might be of use.

Also of interest is this page in the Cloud9 documentation which looks at how you can deal with an XML dump of Wikipedia in MapReduce.