Both sides previous revision
Previous revision
Next revision
|
Previous revision
Next revision
Both sides next revision
|
courses:mapreduce-tutorial:step-7 [2012/01/25 22:01] straka |
courses:mapreduce-tutorial:step-7 [2012/01/28 15:33] majlis |
| |
A cluster can be created using | A cluster can be created using |
/home/straka/hadoop/bin/hadoop-cluster -c number_of_machines -w sec_to_run_the_cluster_for | /net/projects/hadoop/bin/hadoop-cluster -c number_of_machines -w sec_to_run_the_cluster_for |
The syntax is the same as in ''perl script.pl run''. | The syntax is the same as in ''perl script.pl run''. |
| |
===== Example ===== | ===== Example ===== |
| |
Try running the same script {{:courses:mapreduce-tutorial:step-6.txt|wordcount.pl}} as in the last step, this time by creating the cluster and submitting the job to it: | Try running the same script {{:courses:mapreduce-tutorial:step-6.txt|step-7-wordcount.pl}} as in the last step, this time by creating the cluster and submitting the job to it: |
/home/straka/hadoop/bin/hadoop-cluster -c 1 -w 600 | wget --no-check-certificate 'https://wiki.ufal.ms.mff.cuni.cz/_media/courses:mapreduce-tutorial:step-6.txt' -O 'step-7-wordcount.pl' |
perl wordcount.pl run -jt cluster_master:9001 -Dmapred.max.split.size=1000000 /home/straka/wiki/cs-text-medium some_output_directory | /net/projects/hadoop/bin/hadoop-cluster -c 1 -w 600 |
| rm -rf step-7-out-sol; perl step-7-wordcount.pl run -jt cluster_master:9001 -Dmapred.max.split.size=1000000 /home/straka/wiki/cs-text-medium step-7-out-sol |
| less less step-7-out-sol/part-* |
| |
| |
| ---- |
| |
| <html> |
| <table style="width:100%"> |
| <tr> |
| <td style="text-align:left; width: 33%; "></html>[[step-6|Step 6]]: Running on cluster.<html></td> |
| <td style="text-align:center; width: 33%; "></html>[[.|Overview]]<html></td> |
| <td style="text-align:right; width: 33%; "></html>[[step-8|Step 8]]: Multiple mappers, reducers and partitioning.<html></td> |
| </tr> |
| </table> |
| </html> |