[ Skip to the content ]

Institute of Formal and Applied Linguistics Wiki


[ Back to the navigation ]

This is an old revision of the document!


Table of Contents

MapReduce Tutorial : Custom sorting and grouping comparators.

Fast sorting comparator

The keys are sorted before processed by a reducer, using a
Raw comparator. The default comparator uses the compareTo method provided by the key type, which is a subclass of WritableComparable. Consider for example the following IntPair type:

public static class IntPair implements WritableComparable<IntPair> {
  private int first = 0;
  private int second = 0;
 
  public void set(int left, int right) { first = left; second = right; }
  public int getFirst() { return first; }
  public int getSecond() { return second; }
 
  public void readFields(DataInput in) throws IOException {
    first = in.readInt();
    second = in.readInt();
  }
  public void write(DataOutput out) throws IOException {
    out.writeInt(first);
    out.writeInt(second);
  }
 
  public int compareTo(IntPair o) {
    if (first != o.first) return first < o.first ? -1 : 1;
    else return second < o.second ? -1 : second == o.second ? 0 : 1;
  }
}

If we would like in a Hadoop job to sort the IntPair using the first element only, we can provide a RawComparator and set it using job.setSortComparatorClass:

Grouping comparator

In a reduce, it is guaranteed that keys are processed in ascending order. Sometimes it would be useful if the values associated with one key could also be processed in ascending order.


Step 28: Custom data types. Overview Step 30: Custom input formats.


[ Back to the navigation ] [ Back to the content ]