[ Skip to the content ]

Institute of Formal and Applied Linguistics Wiki

[ Back to the navigation ]

You are here: start » courses » mapreduce-tutorial » step-7

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
courses:mapreduce-tutorial:step-7 [2012/01/29 20:52]
straka 		courses:mapreduce-tutorial:step-7 [2013/02/08 14:36] (current)
popel Milan improved our Hadoop
Line 4: Line 4:
  
 A cluster can be created using A cluster can be created using
-  /net/projects/hadoop/bin/hadoop-cluster -c number_of_machines -w sec_to_run_the_cluster_for+  /net/projects/hadoop/bin/hadoop-cluster -c number_of_machines -w sec_to_wait_after_all_jobs_completed
 The syntax is the same as in ''perl script.pl run''. The syntax is the same as in ''perl script.pl run''.
  
Line 11: Line 11:
 ===== Using a running cluster ===== ===== Using a running cluster =====
 Running cluster is identified by its master. When running a Hadoop job using Perl API, existing cluster can be used by Running cluster is identified by its master. When running a Hadoop job using Perl API, existing cluster can be used by
-  perl script.pl run -jt cluster_master:9001 ...+  perl script.pl -jt cluster_master:9001 ... 
 + 
 +===== Running Hadoop jobs from now on ===== 
 + 
 +From now on, it is best to run MR jobs using a one-machine cluster -- create a one-machine cluster using ''hadoop-cluster'' for 3h (10800s) and run jobs using ''-jt cluster_master''. Running the scripts locally without any cluster has several disadvantages, most notably having only one reducer per job. 
  
 ===== Example ===== ===== Example =====
Line 18: Line 22:
   wget --no-check-certificate 'https://wiki.ufal.ms.mff.cuni.cz/_media/courses:mapreduce-tutorial:step-6.txt' -O 'step-7-wordcount.pl'   wget --no-check-certificate 'https://wiki.ufal.ms.mff.cuni.cz/_media/courses:mapreduce-tutorial:step-6.txt' -O 'step-7-wordcount.pl'
   /net/projects/hadoop/bin/hadoop-cluster -c 1 -w 600   /net/projects/hadoop/bin/hadoop-cluster -c 1 -w 600
-  rm -rf step-7-out-sol; perl step-7-wordcount.pl run -jt cluster_master:9001 -Dmapred.max.split.size=1000000 /home/straka/wiki/cs-text-medium step-7-out-sol+  # NOW VIEW THE FILE 
 +  # $EDITOR step-7-wordcount.pl 
 +  rm -rf step-7-out-sol; perl step-7-wordcount.pl -jt cluster_master:9001 -Dmapred.max.split.size=1000000 /home/straka/wiki/cs-text-medium step-7-out-sol
   less less step-7-out-sol/part-*   less less step-7-out-sol/part-*
 Remarks: Remarks:
   * The reducers seem to start running before the mappers finish. In the web interface, the running time of reducers is divided into thirds:   * The reducers seem to start running before the mappers finish. In the web interface, the running time of reducers is divided into thirds:
-    * during the first 33%, the mapper outputs are copied to the machine where reducer runs +    * during the first 33%, the mapper outputs are copied to the machine where reducer runs. 
-    * during the second 33%, the (key, value) pairs are sorted +    * during the second 33%, the (key, value) pairs are sorted. 
-    * during the last 33%, the user-defined reducer runs +    * during the last 33%, the user-defined reducer runs.
  
 ---- ----

[ Back to the navigation ] [ Back to the content ]