Both sides previous revision
Previous revision
Next revision
|
Previous revision
Next revision
Both sides next revision
|
courses:mapreduce-tutorial:step-12 [2012/01/25 21:24] straka |
courses:mapreduce-tutorial:step-12 [2012/01/29 21:14] straka |
Sometimes it would be useful to create output files manually in reducers -- either multiple files are needed per reducer, or a specific file format is desired. | Sometimes it would be useful to create output files manually in reducers -- either multiple files are needed per reducer, or a specific file format is desired. |
| |
Problem is that Hadoop framework can spawn same reducer multiple times -- either because of speculative execution, or if one reducer is presumed to have crashed, even if it in fact did not. | Problem is that Hadoop framework can spawn several task attempts for the same reducer task -- either because of speculative execution, or if one reduce attempt is presumed to have crashed, even if it in fact did not. |
| |
For these reasons Hadoop creates an output directory for every reduce attempt it makes. If the reducer finishes successfully, the files in this directory are moved to the output directory. Still, user must ensure different reducers produce different filenames, usually by naming the files using the serial number of reducer. | For these reasons Hadoop creates an output directory for every reduce attempt it makes. If the reducer finishes successfully, the files in this directory are moved to the output directory. Still, user must ensure different reducers produce different filenames, usually by naming the files using the serial number of reducer. |
Both these informations are available in Perl API using environmental variables: | Both these informations are available in Perl API using environmental variables: |
* ''HADOOP_TASK_ID'' -- available in every mapper and reducer. The serial number of the mapper and reducer task (in range 0..number_of_tasks-1). | * ''HADOOP_TASK_ID'' -- available in every mapper and reducer. The serial number of the mapper and reducer task (in range 0..number_of_tasks-1). |
* ''HADOOP_WORK_OUTPUT_PATH'' | * ''HADOOP_WORK_OUTPUT_PATH'' -- available in a reducer. It contains an existing directory where the reducer can output files. If the reducer finishes successfully, all files and subdirectories will be moved to output directory of the job. |
| |
| ===== Reduce-less jobs ===== |
| If a MR job runs without reducers, the output of mappers is written to output directory without further processing. In this case, environmental variable ''HADOOP_WORK_OUTPUT_PATH'' is present even in a mapper and the files created in this directory are copied to the job output directory. |
| |
| |
| ===== Exercise ===== |
| Change the word counting script {{:courses:mapreduce-tutorial:step-5-solution1.txt|step-12-exercise.pl}} to produce results in reducers manually using the mentioned environmental variables, and execute it using four reducers. |
| |
| wget --no-check-certificate 'https://wiki.ufal.ms.mff.cuni.cz/_media/courses:mapreduce-tutorial:step-5-solution1.txt' -O 'step-12-exercise.pl' |
| rm -rf step-12-out-ex; perl step-12-exercise.pl run -c 5 -r 4 /home/straka/wiki/cs-text-medium/ step-12-out-ex |
| less step-12-out-ex/part-* |
| |
| ==== Solution ==== |
| You can also download the solution {{:courses:mapreduce-tutorial:step-12-solution.txt|step-12-solution.pl}} and check the correct output. |
| wget --no-check-certificate 'https://wiki.ufal.ms.mff.cuni.cz/_media/courses:mapreduce-tutorial:step-12-solution.txt' -O 'step-12-solution.pl' |
| rm -rf step-12-out-sol; perl step-12-solution.pl run -c 5 - r 4 /home/straka/wiki/cs-text-medium/ step-12-out-sol |
| less step-12-out-sol/vystup-* |
| |
| ---- |
| |
| <html> |
| <table style="width:100%"> |
| <tr> |
| <td style="text-align:left; width: 33%; "></html>[[step-11|Step 11]]: Initialization and cleanup of MR tasks, performance of combiners.<html></td> |
| <td style="text-align:center; width: 33%; "></html>[[.|Overview]]<html></td> |
| <td style="text-align:right; width: 33%; "></html>[[step-13|Step 13]]: Sorting.<html></td> |
| </tr> |
| </table> |
| </html> |
| |