Both sides previous revision
Previous revision
Next revision
|
Previous revision
Next revision
Both sides next revision
|
courses:mapreduce-tutorial:step-8 [2012/01/29 21:12] straka |
courses:mapreduce-tutorial:step-8 [2012/01/31 09:36] straka The number of reducers must be specified in the exercise. |
| |
<code perl> | <code perl> |
package Partitioner; | package My::Partitioner; |
use Moose; | use Moose; |
with 'Hadoop::Partitioner'; | with 'Hadoop::Partitioner'; |
| |
... | ... |
package Main; | package main; |
use Hadoop::Runner; | use Hadoop::Runner; |
| |
my $runner = Hadoop::Runner->new( | my $runner = Hadoop::Runner->new( |
... | ... |
partitioner => Partitioner->new(), | partitioner => My::Partitioner->new(), |
...); | ...); |
... | ... |
Run one MR job on '/home/straka/wiki/cs-text-medium', which creates two output files -- one with ascending list of unique article names and the other with an ascending list of unique words. You can download the template {{:courses:mapreduce-tutorial:step-8-exercise.txt|step-8-exercise.pl}} and execute it. | Run one MR job on '/home/straka/wiki/cs-text-medium', which creates two output files -- one with ascending list of unique article names and the other with an ascending list of unique words. You can download the template {{:courses:mapreduce-tutorial:step-8-exercise.txt|step-8-exercise.pl}} and execute it. |
wget --no-check-certificate 'https://wiki.ufal.ms.mff.cuni.cz/_media/courses:mapreduce-tutorial:step-8-exercise.txt' -O 'step-8-exercise.pl' | wget --no-check-certificate 'https://wiki.ufal.ms.mff.cuni.cz/_media/courses:mapreduce-tutorial:step-8-exercise.txt' -O 'step-8-exercise.pl' |
rm -rf step-8-out-ex; perl step-8-exercise.pl run /home/straka/wiki/cs-text-medium/ step-8-out-ex | # NOW EDIT THE FILE |
| # $EDITOR step-8-exercise.pl |
| rm -rf step-8-out-ex; perl step-8-exercise.pl -c 2 -r 2 /home/straka/wiki/cs-text-medium/ step-8-out-ex |
less step-8-out-ex/part-* | less step-8-out-ex/part-* |
| |
You can also download the solution {{:courses:mapreduce-tutorial:step-8-solution.txt|step-8-solution.pl}} and check the correct output. | You can also download the solution {{:courses:mapreduce-tutorial:step-8-solution.txt|step-8-solution.pl}} and check the correct output. |
wget --no-check-certificate 'https://wiki.ufal.ms.mff.cuni.cz/_media/courses:mapreduce-tutorial:step-8-solution.txt' -O 'step-8-solution.pl' | wget --no-check-certificate 'https://wiki.ufal.ms.mff.cuni.cz/_media/courses:mapreduce-tutorial:step-8-solution.txt' -O 'step-8-solution.pl' |
rm -rf step-8-out-sol; perl step-8-solution.pl run /home/straka/wiki/cs-text-medium/ step-8-out-sol | # NOW VIEW THE FILE |
| # $EDITOR step-8-solution.pl |
| rm -rf step-8-out-sol; perl step-8-solution.pl -c 2 -r 2 /home/straka/wiki/cs-text-medium/ step-8-out-sol |
less step-8-out-sol/part-* | less step-8-out-sol/part-* |
| |