Differences
This shows you the differences between two versions of the page.
Next revision | Previous revision Next revision Both sides next revision | ||
courses:mapreduce-tutorial:step-10 [2012/01/25 15:46] straka vytvořeno |
courses:mapreduce-tutorial:step-10 [2012/01/25 19:33] straka |
||
---|---|---|---|
Line 1: | Line 1: | ||
- | ====== MapReduce Tutorial : ====== | + | ====== MapReduce Tutorial : Combiners ====== |
+ | |||
+ | Sometimes the reduce is a binary operation, which is associative and commutative, | ||
+ | |||
+ | Instead, reducer can be executed right after the map, on //some portion// of values belonging to the same key. Only the results are then sent through the network. | ||
+ | |||
+ | A Hadoop job can have such locally executed reducer, called // | ||
+ | |||
+ | Typically, the combiner is the same as the reducer of a MR job. | ||
+ | |||
+ | <code perl> | ||
+ | package Mapper; | ||
+ | ... | ||
+ | |||
+ | package Reducer; | ||
+ | ... | ||
+ | |||
+ | package Main; | ||
+ | use Hadoop:: | ||
+ | |||
+ | my $runner = Hadoop:: | ||
+ | mapper => Mapper-> | ||
+ | combiner => Reducer-> | ||
+ | reducer => Reducer-> | ||
+ | input_format => ' | ||
+ | ... | ||
+ | </ | ||
+ | |||
+ | ===== Excersise | ||
+ | |||
+ | Compare the effect of adding the combiner to a MR job which counts occurences of words of ''/ | ||
+ | |||
+ | How would you explain the results? | ||
+ | |||
+ | {{: |