hadoop - map 减少链接

这个问题已经在这里有了答案:




已关闭8年。






我有 map 减少链如下所述。

Job1(Map1-> Reduce 1)-> Job2(Map2，Reduce2)Job1.waitForCompletion(true)

我需要Map2中的一个值(假设int a是由Reduce 1创建的)。

我怎样才能做到这一点 ？？请分享您的想法

最佳答案

您可以使用ChainMapper和ChainReducer。这是帮助您的示例代码。

Configuration conf = getConf();
JobConf job = new JobConf(conf);
JobConf Conf1 = new JobConf(false);
ChainMapper.setMapper
(job,
Map1.class,
LongWritable.class,
Text.class,
Text.class,
Text.class,
true,
Conf1);


 JobConf Conf2 = new JobConf(false);
    ChainReducer.setReducer
    (job,
    Reduce1.class,
    Text.class,
    Text.class,
    Text.class,
    Text.class,
    true,
    Conf2);
JobConf Conf3 = new JobConf(false);
    ChainMapper.setMapper
    (job,
    Map2.class,
    Text.class,
    Text.class,
    Text.class,
    Text.class,
    true,
    Conf3);
JobConf Conf4 = new JobConf(false);
    ChainReducer.setReducer
    (job,
    Reduce2.class,
    Text.class,
    Text.class,
    Text.class,
    Text.class,
    true,
    Conf4);

注意:

the out-put Type of  key-value derive which Mapper and reducer is to be called next so , the output Type of Map1 should me same as Input Type of key-value of Reduce1 AND the output Type of Reduce1 should me same as Input Type of key-value of Map2 and
the output Type of Map2 should me same as Input Type of key-value of Reduce2

关于hadoop - map 减少链接，我们在Stack Overflow上找到一个类似的问题：https://stackoverflow.com/questions/13193150/