[ Skip to the content ]

Institute of Formal and Applied Linguistics Wiki

[ Back to the navigation ]

You are here: start » courses » mapreduce-tutorial » step-3

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
Next revision Both sides next revision
courses:mapreduce-tutorial:step-3 [2012/01/24 21:11]
straka 		courses:mapreduce-tutorial:step-3 [2012/01/28 11:09]
majlis Odstraneny soubory s priklady
Line 1: Line 1:
 ====== MapReduce Tutorial : Basic mapper ====== ====== MapReduce Tutorial : Basic mapper ======
  
-The simplest MR job consists of a mapper only.  The input data is divided in several parts, every processed by an independent mapper, and the results are collected in one directory, one file per mapper.+The simplest Hadoop job consists of a mapper only.  The input data is divided in several parts, every processed by an independent mapper, and the results are collected in one directory, one file per mapper. 
 + 
 +The Hadoop framework silently handles failures. If a mapper task fails, another is executed and the input of the failed attempt is discarded.
  
 ===== Example Perl mapper ===== ===== Example Perl mapper =====
  
-<file perl mapper.pl>+<file perl>
 #!/usr/bin/perl #!/usr/bin/perl
  
Line 32: Line 34:
 The values ''input_format'', ''output_format'' and ''output_compression'' could be left out, because they are all set to their default value. The values ''input_format'', ''output_format'' and ''output_compression'' could be left out, because they are all set to their default value.
  
-Resulting script can be executed locally (not distributed) using+Resulting script can be executed locally in a single thread using
   perl script.pl run input_directory output_directory   perl script.pl run input_directory output_directory
 All files in input_directory are processes. The output_directory must not exist. All files in input_directory are processes. The output_directory must not exist.
Line 38: Line 40:
 ===== Exercise ===== ===== Exercise =====
  
-To check that your Hadoop environment works, try running a MR job on ''/home/straka/wiki/cs-text'', which outputs only articles with names beginning with a (ignoring the case).+To check that your Hadoop environment works, try running a MR job on ''/home/straka/wiki/cs-text'', which outputs only articles with names beginning with an ''A'' (ignoring the case). You can download the template  and execute it using 
 +  wget --no-check-certificate 'https://wiki.ufal.ms.mff.cuni.cz/_media/courses:mapreduce-tutorial:step-3-exercise.txt' -O 'step-3-exercise.pl' 
 +  rm -rf step-3-output; perl step-3-exercise.pl run /home/straka/wiki/cs-text step-3-output 
  
-{{.:step-3:exercise.txt|Solution.pl}} 

[ Back to the navigation ] [ Back to the content ]