Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
courses:mapreduce-tutorial:step-10 [2012/01/25 18:37] straka |
courses:mapreduce-tutorial:step-10 [2012/01/25 19:30] straka |
||
---|---|---|---|
Line 1: | Line 1: | ||
====== MapReduce Tutorial : Combiners ====== | ====== MapReduce Tutorial : Combiners ====== | ||
+ | Sometimes the reduce is a binary operation, which is associative and commutative, | ||
+ | Instead, reducer can be executed right after the map, on //some portion// of values belonging to the same key. Only the results are then sent through the network. | ||
+ | |||
+ | A Hadoop job can have such locally executed reducer, called // | ||
+ | |||
+ | Typically, the combiner is the same as the reducer of a MR job. | ||
+ | |||
+ | <code perl> | ||
+ | package Mapper; | ||
+ | ... | ||
+ | |||
+ | package Reducer; | ||
+ | ... | ||
+ | |||
+ | package Main; | ||
+ | use Hadoop:: | ||
+ | |||
+ | my $runner = Hadoop:: | ||
+ | mapper => Mapper-> | ||
+ | combiner => Reducer-> | ||
+ | reducer => Reducer-> | ||
+ | input_format => ' | ||
+ | ... | ||
+ | </ | ||
+ | |||
+ | ===== Excersise ===== | ||
+ | |||
+ | Compare the effect of adding the combiner to a MR job which counts occurences of words of ''/ | ||
+ | |||
+ | How would you explain the results? |