Differences

This shows you the differences between two versions of the page.

--- courses:mapreduce-tutorial:running-jobs [2012/02/05 21:21]
straka
+++ courses:mapreduce-tutorial:running-jobs [2013/02/08 14:33] (current)
popel Milan improved our Hadoop
@@ Line 8: / Line 8: @@
 | ^ Command ^
-^ Run Perl script ''script.pl'' | ''perl script.pl'' options |
+^ Run Perl script ''script.pl'' | ''perl script.pl'' //options// |
-^ Run Java job ''job.jar'' | ''/net/projects/hadoop/bin/hadoop job.jar'' options |
+^ Run Java job ''job.jar'' | ''/net/projects/hadoop/bin/hadoop job.jar'' //options// |
 The options are the same for Perl and java:
@@ Line 21: / Line 21: @@
 ^ Run using //M// mappers | ''`/net/projects/hadoop/bin/compute-splitsize input M` input output'' |
+From February 2012, using the parameter ''-w'' makes Hadoop to wait W seconds after the **last** task is finished. This means that you can start a cluster for one task (with ''-c N_machines -w W_seconds'') and reuse it (with ''-jt jobtracker:port'') for other tasks without worries that the other tasks will be killed before finishing.
+===== Running multiple jobs =====
+There are several ways of running multiple jobs:
+  * Java only: Create multiple ''Job'' instances and call ''submit'' or ''waitForCompletion'' multiple times
+  * Create a cluster using ''/net/projects/hadoop/bin/hadoop-cluster'', parse the jobtracker:port using ''head -1'' and run the jobs using ''-jt jobtracker:port''
+  * Create a shell script running multiple jobs using ''-jt HADOOP_JOBTRACKER''. Then run it using ''//net/projects/hadoop/bin/hadoop-cluster -c machines script.sh''.

[ Back to the navigation ] [ Back to the content ]

Institute of Formal and Applied Linguistics Wiki

Differences