Differences
This shows you the differences between two versions of the page.
Next revision | Previous revision Next revision Both sides next revision | ||
courses:mapreduce-tutorial:step-12 [2012/01/25 15:46] straka vytvořeno |
courses:mapreduce-tutorial:step-12 [2012/01/25 21:29] straka |
||
---|---|---|---|
Line 1: | Line 1: | ||
- | ====== MapReduce Tutorial : ====== | + | ====== MapReduce Tutorial : Additional output from mappers and reducers ====== |
+ | |||
+ | Sometimes it would be useful to create output files manually in reducers -- either multiple files are needed per reducer, or a specific file format is desired. | ||
+ | |||
+ | Problem is that Hadoop framework can spawn several task attempts for the same reducer task -- either because of speculative execution, or if one reduce attempt is presumed to have crashed, even if it in fact did not. | ||
+ | |||
+ | For these reasons Hadoop creates an output directory for every reduce attempt it makes. If the reducer finishes successfully, | ||
+ | |||
+ | Both these informations are available in Perl API using environmental variables: | ||
+ | * '' | ||
+ | * '' | ||
+ | |||
+ | ===== Reduce-less jobs ===== | ||
+ | If a MR job runs without reducers, the output of mappers is written to output directory without further processing. In this case, environmental variable '' |