反向分类减速键

问题描述

将地图输出键以相反顺序放到缩小器的最佳方法是什么？默认情况下，减速器以按键的升序接收所有的键。任何帮助或意见广泛赞赏。简单的说，在正常情况下，如果一张地图发出键 1,4,3,5,2 ，缩减器会收到相同的作为 1,2,3,4,5 。我想减速机接受 5,4,3,2,1 。

解决方案

在Hadoop 1.X中，您可以使用。

您的比较器必须实现。

使用Hadoop 2.X，可以使用，执行 RawComparator 。

What is the best approach to get the Map Output keys to a reducer in reverse order? By default the reducer receives all keys in ascending order of keys. Any help or comments widely appreciated.

In simple words, in the normal scenario, if a map emits keys 1,4,3,5,2 the reducer receives the same as 1,2,3,4,5. I would like the reducer to receive 5,4,3,2,1 instead.

解决方案

In Hadoop 1.X, you can specify a custom comparator class for your outputs using JobConf.setOutputKeyComparatorClass.

Your comparator must implement the RawComparator interface.

With Hadoop 2.X, this is done by using Job.setSortComparatorClass, still with an implementation of RawComparator.

这篇关于反向分类减速键的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！