[ Skip to the content ]

Institute of Formal and Applied Linguistics Wiki

[ Back to the navigation ]

You are here: start » courses » mapreduce-tutorial » step-7

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
Next revision Both sides next revision
courses:mapreduce-tutorial:step-7 [2012/01/29 20:51]
straka 		courses:mapreduce-tutorial:step-7 [2012/01/29 23:52]
majlis
Line 18: Line 18:
   wget --no-check-certificate 'https://wiki.ufal.ms.mff.cuni.cz/_media/courses:mapreduce-tutorial:step-6.txt' -O 'step-7-wordcount.pl'   wget --no-check-certificate 'https://wiki.ufal.ms.mff.cuni.cz/_media/courses:mapreduce-tutorial:step-6.txt' -O 'step-7-wordcount.pl'
   /net/projects/hadoop/bin/hadoop-cluster -c 1 -w 600   /net/projects/hadoop/bin/hadoop-cluster -c 1 -w 600
 +  # NOW VIEW THE FILE
 +  # $EDITOR step-7-wordcount.pl
   rm -rf step-7-out-sol; perl step-7-wordcount.pl run -jt cluster_master:9001 -Dmapred.max.split.size=1000000 /home/straka/wiki/cs-text-medium step-7-out-sol   rm -rf step-7-out-sol; perl step-7-wordcount.pl run -jt cluster_master:9001 -Dmapred.max.split.size=1000000 /home/straka/wiki/cs-text-medium step-7-out-sol
   less less step-7-out-sol/part-*   less less step-7-out-sol/part-*
 Remarks: Remarks:
-  * The reducers seem to start running before the mappers finishes. In the web interface, the running time of reducers is divided into thirds: during the first 33%, the mapper outputs are copied +  * The reducers seem to start running before the mappers finish. In the web interface, the running time of reducers is divided into thirds: 
 +    * during the first 33%, the mapper outputs are copied to the machine where reducer runs. 
 +    * during the second 33%, the (key, value) pairs are sorted. 
 +    * during the last 33%, the user-defined reducer runs.
  
 ---- ----

[ Back to the navigation ] [ Back to the content ]