MapReduce 编程 系列九 Reducer数目

本篇介绍怎样控制reduce的数目。前面观察结果文件，都会发现通常是以part-r-00000 形式出现多个文件，事实上这个reducer的数目有关系。reducer数目多，结果文件数目就多。

在初始化job的时候。是能够设置reducer的数目的。example4在example的基础上做了改动。改动了pom.xml。使得结束一个參数作为reducer的数目。改动了LogJob.java的代码，作为设置reducer数目。

         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">

  <modelVersion>4.0.0</modelVersion>

  <groupId>org.freebird</groupId>

  <artifactId>mr1_example4</artifactId>

  <packaging>jar</packaging>

  <version>1.0-SNAPSHOT</version>

  <name>mr1_example4</name>

  <url>http://maven.apache.org</url>

  <dependencies>

    <dependency>

      <groupId>org.apache.hadoop</groupId>

      <artifactId>hadoop-core</artifactId>

      <version>1.2.1</version>

    </dependency>

  </dependencies>

  <build>

    <plugins>

      <plugin>

        <groupId>org.codehaus.mojo</groupId>

        <artifactId>exec-maven-plugin</artifactId>

        <version>1.3.2</version>

        <executions>

          <execution>

            <goals>

              <goal>exec</goal>

            </goals>

          </execution>

        </executions>

        <configuration>

          <executable>hadoop</executable>

          <arguments>

            <argument>jar</argument>

            <argument>target/mr1_example4-1.0-SNAPSHOT.jar</argument>

            <argument>org.freebird.LogJob</argument>

            <argument>/user/chenshu/share/logs</argument>

            <argument>1</argument>

          </arguments>

        </configuration>

      </plugin>

    </plugins>

  </build>

</project>

LogJob.java代码:

import org.apache.hadoop.io.IntWritable;

import org.apache.hadoop.fs.Path;

import org.apache.hadoop.conf.Configuration;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Job;

import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;

import org.freebird.reducer.LogReducer;

import org.freebird.mapper.LogMapper;

import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;

import org.apache.hadoop.mapreduce.lib.output.MultipleOutputs;

import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

import org.apache.hadoop.fs.FileSystem;

import java.io.IOException;

public class LogJob {

    public static void main(String[] args) throws Exception {

        String inputPath = args[0];

        if (inputPath.endsWith("/")) {

            inputPath = inputPath.substring(0, inputPath.length() -1);

        }

        System.out.println("args[0] indicates input folder path, the last / will be removed if it exists:" + inputPath);

        String outputPath = inputPath + "/output";

        System.out.println("output folder path is:" + outputPath);

        int numReducer = Integer.parseInt(args[1]);

        System.out.println("reducer number is: " + args[1]);

        Configuration conf = new Configuration();

        Job job = new Job(conf, "sum_did_from_log_file");

        job.setJarByClass(LogJob.class);

        job.setMapperClass(org.freebird.mapper.LogMapper.class);

        job.setReducerClass(org.freebird.reducer.LogReducer.class);

        job.setNumReduceTasks(numReducer);

        job.setOutputKeyClass(Text.class);

        job.setOutputValueClass(IntWritable.class);

        Path path1 = new Path(inputPath);

        Path path2 = new Path(outputPath);

        removeFolder(path2, conf);

        MultipleOutputs.addNamedOutput(job, "result", TextOutputFormat.class, Text.class, IntWritable.class);

        FileInputFormat.addInputPath(job, path1);

        FileOutputFormat.setOutputPath(job, path2);

        System.exit(job.waitForCompletion(true) ? 0 : 1);

    }

    private static void removeFolder(Path path, Configuration conf) throws IOException {

        FileSystem fs = path.getFileSystem(conf);

        if (fs.exists(path)) {

            fs.delete(path);

        }

    }

}

执行结果，通过观察jobtracker。的确reducer数目为1了。

而且结果文件也变成了仅仅有一个：

[chenshu@hadoopMaster example4]$ hdfs dfs -ls /user/chenshu/share/logs/output/

14/10/03 14:18:35 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Found 4 items

-rw-r--r--   3 chenshu chenshu          0 2014-10-03 12:53 /user/chenshu/share/logs/output/_SUCCESS

drwxr-xr-x   - chenshu chenshu          0 2014-10-03 12:52 /user/chenshu/share/logs/output/_logs

-rw-r--r--   3 chenshu chenshu          0 2014-10-03 12:53 /user/chenshu/share/logs/output/part-r-00000

-rw-r--r--   3 chenshu chenshu    4391668 2014-10-03 12:53 /user/chenshu/share/logs/output/result-r-00000