Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
courses:mapreduce-tutorial:step-3 [2012/01/24 19:14] straka |
courses:mapreduce-tutorial:step-3 [2012/01/31 09:40] (current) straka Change Perl commandline syntax. |
||
---|---|---|---|
Line 1: | Line 1: | ||
====== MapReduce Tutorial : Basic mapper ====== | ====== MapReduce Tutorial : Basic mapper ====== | ||
- | The simplest | + | The simplest |
- | ===== Example Perl mapper | + | The Hadoop framework silently handles failures. If a mapper |
- | <code perl> | + | ===== Example Perl mapper ===== |
- | # | + | |
- | package Mapper; | + | <file perl> |
+ | package | ||
use Moose; | use Moose; | ||
with ' | with ' | ||
Line 18: | Line 18: | ||
} | } | ||
- | package | + | package |
use Hadoop:: | use Hadoop:: | ||
my $runner = Hadoop:: | my $runner = Hadoop:: | ||
- | mapper => Mapper-> | + | mapper => My::Mapper-> |
input_format => ' | input_format => ' | ||
output_format => ' | output_format => ' | ||
output_compression => 0); | output_compression => 0); | ||
- | $runner-> | + | $runner-> |
- | </code> | + | </file> |
The values '' | The values '' | ||
- | Resulting script can be executed using | + | Resulting script can be executed |
- | perl script.pl | + | perl script.pl |
- | All files in input_directory | + | Input can be either file or a directory -- in that case, all files in this directory |
+ | |||
+ | === Standard input and output === | ||
+ | Standard input and standard output of the Perl script are used to communicate with the Hadoop framework. Therefore you should use standard error output if you want to print some debugging output using '' | ||
+ | |||
+ | ===== Exercise ===== | ||
+ | |||
+ | To check that your Hadoop environment works, try running a MR job on ''/ | ||
+ | wget --no-check-certificate ' | ||
+ | # NOW EDIT THE FILE | ||
+ | # $EDITOR step-3-exercise.pl | ||
+ | rm -rf step-3-out-ex; | ||
+ | less step-3-out-ex/ | ||
+ | |||
+ | ==== Solution ==== | ||
+ | You can also download the solution {{: | ||
+ | wget --no-check-certificate ' | ||
+ | # NOW VIEW THE FILE | ||
+ | # $EDITOR step-3-solution.pl | ||
+ | rm -rf step-3-out-sol; | ||
+ | less step-3-out-sol/ | ||
+ | |||
+ | ---- | ||
+ | |||
+ | < | ||
+ | <table style=" | ||
+ | < | ||
+ | <td style=" | ||
+ | <td style=" | ||
+ | <td style=" | ||
+ | </ | ||
+ | </ | ||
+ | </ |