Both sides previous revision
Previous revision
Next revision
|
Previous revision
Next revision
Both sides next revision
|
courses:mapreduce-tutorial:step-12 [2012/01/28 12:25] majlis |
courses:mapreduce-tutorial:step-12 [2012/01/29 14:56] majlis |
===== Reduce-less jobs ===== | ===== Reduce-less jobs ===== |
If a MR job runs without reducers, the output of mappers is written to output directory without further processing. In this case, environmental variable ''HADOOP_WORK_OUTPUT_PATH'' is present even in a mapper and the files created in this directory are copied to the job output directory. | If a MR job runs without reducers, the output of mappers is written to output directory without further processing. In this case, environmental variable ''HADOOP_WORK_OUTPUT_PATH'' is present even in a mapper and the files created in this directory are copied to the job output directory. |
| |
| |
===== Exercise ===== | ===== Exercise ===== |
Change the word counting script {{:courses:mapreduce-tutorial:TODOstep-5-solution1.txt|TODOwc.pl}} to produce results in reducers manually using the mentioned environmental variables, and execute it using four reducers. | Change the word counting script {{:courses:mapreduce-tutorial:step-5-solution1.txt|step-12-exercise.pl}} to produce results in reducers manually using the mentioned environmental variables, and execute it using four reducers. |
| |
| wget --no-check-certificate 'https://wiki.ufal.ms.mff.cuni.cz/_media/courses:mapreduce-tutorial:step-5-solution1.txt' -O 'step-12-exercise.pl' |
| rm -rf step-12-out-ex; perl step-12-exercise.pl run -c 5 - r 4 /home/straka/wiki/cs-text-medium/ step-12-out-ex |
| less step-12-out-ex/part-* |
| |
| ==== Solution ==== |
| You can also download the solution {{:courses:mapreduce-tutorial:step-12-solution.txt|step-12-solution.pl}} and check the correct output. |
| wget --no-check-certificate 'https://wiki.ufal.ms.mff.cuni.cz/_media/courses:mapreduce-tutorial:step-12-solution.txt' -O 'step-12-solution.pl' |
| rm -rf step-12-out-sol; perl step-12-solution.pl run -c 5 - r 4 /home/straka/wiki/cs-text-medium/ step-12-out-sol |
| less step-12-out-sol/vystup-* |
| |
| ---- |
| |
{{:courses:mapreduce-tutorial:step-12-solution.txt|Solution.pl}} | <html> |
| <table style="width:100%"> |
| <tr> |
| <td style="text-align:left; width: 33%; "></html>[[step-11|Step 11]]: Initialization and cleanup of MR tasks, performance of combiners.<html></td> |
| <td style="text-align:center; width: 33%; "></html>[[.|Overview]]<html></td> |
| <td style="text-align:right; width: 33%; "></html>[[step-13|Step 13]]: Sorting.<html></td> |
| </tr> |
| </table> |
| </html> |
| |