1. Combiner概述

Hadoop(19)-MapReduce框架原理-Combiner合并-LMLPHP

2. 自定义Combiner实现步骤

1). 定义一个Combiner继承Reducer,重写reduce方法

public class WordcountCombiner extends Reducer<Text, IntWritable, Text,IntWritable>{

    @Override
protected void reduce(Text key, Iterable<IntWritable> values,Context context) throws IOException, InterruptedException { // 1 汇总操作
int count = 0;
for(IntWritable v :values){
count += v.get();
} // 2 写出
context.write(key, new IntWritable(count));
}
}

2). 在Driver类中添加设置

job.setCombinerClass(WordcountCombiner.class);

效果

Hadoop(19)-MapReduce框架原理-Combiner合并-LMLPHP

Hadoop(19)-MapReduce框架原理-Combiner合并-LMLPHP

05-11 22:10
查看更多