且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

hadoop,如何在尝试运行映射作业时包含3part jar

更新时间:2023-09-29 22:10:58

这些被称为通用选项. 因此,为了支持这些功能,您的工作应该实现工具.

Those are called generic options. So, to support those, your job should implement Tool.

像-

hadoop jar yourfile.jar [mainClass] args -libjars <comma seperated list of jars>

要实现 Tool 并扩展 Configured ,您可以在MapReduce应用程序中执行以下操作-

To implement Tool and extend Configured, you do something like this in your MapReduce application --

public class YourClass extends Configured implements Tool {

      public static void main(String[] args) throws Exception {
         int res = ToolRunner.run(new YourClass(), args);
         System.exit(res);
      }

      public int run(String[] args) throws Exception
      {
        //parse you normal arguments here.

        Configuration conf = getConf();
        Job job = new Job(conf, "Name of job");

        //set the class names etc

        //set the output data type classes etc

        //to accept the hdfs input and outpur dir at run time
        FileInputFormat.addInputPath(job, new Path(args[0]));
        FileOutputFormat.setOutputPath(job, new Path(args[1]));

        return job.waitForCompletion(true) ? 0 : 1;
    }
}