[ Skip to the content ]

Institute of Formal and Applied Linguistics Wiki

[ Back to the navigation ]

You are here: start » courses » mapreduce-tutorial » step-9

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision Both sides next revision
courses:mapreduce-tutorial:step-9 [2012/01/25 15:56]
straka 		courses:mapreduce-tutorial:step-9 [2012/01/25 16:07]
straka
Line 5: Line 5:
 The Hadoop itself uses many configuration options. Every option has a (dot-separated) name and a value and can be set on the command line using ''-Dname=value'' syntax: The Hadoop itself uses many configuration options. Every option has a (dot-separated) name and a value and can be set on the command line using ''-Dname=value'' syntax:
   perl script.pl run [-jt cluster_master | -c cluster_size [-w sec_to_wait]] [-r number_of_reducers] [Hadoop options] input_path output_path   perl script.pl run [-jt cluster_master | -c cluster_size [-w sec_to_wait]] [-r number_of_reducers] [Hadoop options] input_path output_path
-Mind that the order of options matters -- the ''-jt'', ''-c'', ''-w'' and ''-r'' must precede Hadoop options. +Mind that the order of options matters -- the ''-jt'', ''-c'', ''-w'' and ''-r'' must precede Hadoop options to be recognized.
  
 +Every Hadoop option has a read-only default. These are overridden by cluster specific options. Lastly, all of these are overriden by job specific options given on the command line (or set using the Java API).
  
 +===== Mapping of Perl options to Hadoop =====
 +^ Perl options ^ Hadoop options ^
 +| no options \\ (running locally) | ''-Dmapred.job.tracker=local'' \\ ''-Dmapred.local.dir=hadoop-localrunner-tmp'' \\ ''-Dhadoop.tmp.dir=hadoop-localrunner-tmp'' |
 +| ''-jt cluster_master'' | ''-Dmapred.job.tracker=cluster_master'' |
 +| ''-c cluster_machines'' | configuration of new cluster contains \\ ''-Dmapred.job.tracker=cluster_master'' |
 +| ''-r number_of_reducers'' | ''-Dmapred.reduce.tasks=number_of_reducers'' |

[ Back to the navigation ] [ Back to the content ]