java - Hadoop mapreduce-java.io.IOException:作业失败

我在尝试执行exception hadoop程序时得到了以下mapreduce。



BookDriver.java

package com.vasa.books;

import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.FileInputFormat;
import org.apache.hadoop.mapred.FileOutputFormat;
import org.apache.hadoop.mapred.JobClient;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapred.TextInputFormat;
import org.apache.hadoop.mapred.TextOutputFormat;

public class BookDriver {

    public static void main(String args[]) {
        // TODO Auto-generated method stub
        JobClient client=new JobClient();
        JobConf conf=new JobConf(com.vasa.books.BookDriver.class);

        conf.setJobName("booknamefind");


        conf.setOutputKeyClass(Text.class);
        conf.setOutputValueClass(IntWritable.class);

        conf.setMapperClass(com.vasa.books.bookmapper.class);
        conf.setReducerClass(com.vasa.books.bookreducer.class);

        conf.setInputFormat(TextInputFormat.class);
        conf.setOutputFormat(TextOutputFormat.class);

        FileInputFormat.setInputPaths(conf,new Path(args[0]));
        FileOutputFormat.setOutputPath(conf, new Path(args[1]));

        client.setConf(conf);
        try{
        JobClient.runJob(conf);
        }catch(Exception e){
        e.printStackTrace();
        }

    }
}

BookMapper.java

package com.vasa.books;

import java.io.IOException;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.io.Writable;
import org.apache.hadoop.io.WritableComparable;
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.Mapper;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reporter;

public class BookMapper extends MapReduceBase implements Mapper<LongWritable,Text,Text,IntWritable> {
    private final static IntWritable one=new IntWritable(1);
    public void map(LongWritable _key, Text value,
            OutputCollector<Text, IntWritable> output, Reporter reporter)
            throws IOException {
        // TODO Auto-generated method stub
        String Tempstring=value.toString();
        String[] singlebookdata=Tempstring.split("\",\"");
        output.collect(new Text(singlebookdata[3]), one);


    }

}

为什么会发生这种异常？

最佳答案

根据JobClient的来源，JobClient.runJob()调用JobClient.monitorAndPrintJob()，它返回一个 bool(boolean) 值。如果该 bool(boolean) 值为假(表示作业失败)，它将打印出无用的错误消息“某事已失败!”。您所看到的。

要解决此问题，您有两种选择:

1-(更快速)检查日志。 RunningJob故障信息应该被打印到日志中。

2-如果您不知道日志在哪里，未启用日志记录或不想翻阅日志，则可以重写一些代码。除了使用JobClient.runJob()之外，我将执行runJob()在您的代码中所做的工作，因此，当失败时，您会收到一条有用的错误消息。

  public static RunningJob myCustomRunJob(JobConf job) throws Exception {
    JobClient jc = new JobClient(job);
    RunningJob rj = jc.submitJob(job);
    if (!jc.monitorAndPrintJob(job, rj)) {
      throw new IOException("Job failed with info: " + rj.getFailureInfo());
    }
    return rj;
  }

我的猜测是潜在的问题是找不到arg [0]或arg [1](您的输入或输出文件)。