The interesting part of a Hadoop job is the reducer – after all mappers produce the (key, value) pairs, for every unique key and all its values a reduce function is called. The reduce function can output (key, value) pairs, which are written to disk.

The reduce is similar to map, but instead of one value it gets an iterator, which enumerates all values associated with the key:

package Mapper;
use Moose;
with 'Hadoop::Mapper';
 
sub map {
  my ($self, $key, $value, $context) = @_;
 
  $context->write($key, $value);
}
 
package Reducer;
use Moose;
with 'Hadoop::Reducer';
 
sub reduce {
  my ($self, $key, $values, $context) = @_;
 
  while ($values->next) {
    $context->write($key, $values->value);
  }
}
 
package Main;
use Hadoop::Runner;
 
my $runner = Hadoop::Runner->new(
  mapper => Mapper->new(),
  reducer => Reducer->new());
 
$runner->run();

As before, Hadoop silently handles failures. It can happen that even a successfully finished mapper needs to be executed again – if the machine, where its output data were stored, gets disconnected from the network.

Run a Hadoop job on /home/straka/wiki/cs-text-small, which counts occurrences of every word in the article texts. You can download the template step-5-exercise1.pl and execute it.

wget --no-check-certificate 'https://wiki.ufal.ms.mff.cuni.cz/_media/courses:mapreduce-tutorial:step-5-exercise1.txt' -O 'step-5-exercise1.pl'
rm -rf step-5-out-ex1; perl step-5-exercise1.pl run /home/straka/wiki/cs-text-medium/ step-5-out-ex1
less step-5-out-ex1/part-*

You can also download the solution step-5-solution1.pl and check the correct output.

wget --no-check-certificate 'https://wiki.ufal.ms.mff.cuni.cz/_media/courses:mapreduce-tutorial:step-5-solution1.txt' -O 'step-5-solution1.pl'
rm -rf step-5-out-sol1; perl step-5-solution1.pl run /home/straka/wiki/cs-text-medium/ step-5-out-sol1
less step-5-out-sol1/part-*

Run a Hadoop job on /home/straka/wiki/cs-text-small, which generates an inverted index. Inverted index contains for each word all its occurrences, where each occurrence is pair (article of occurrence, position of occurrence). You can download the template step-5-exercise2.pl and execute it.

wget --no-check-certificate 'https://wiki.ufal.ms.mff.cuni.cz/_media/courses:mapreduce-tutorial:step-5-exercise2.txt' -O 'step-5-exercise2.pl'
rm -rf step-5-out-ex2; perl step-5-exercise2.pl run /home/straka/wiki/cs-text-tiny/ step-5-out-ex2
less step-5-out-ex2/part-*

You can also download the solution step-5-solution2.pl and check the correct output.

wget --no-check-certificate 'https://wiki.ufal.ms.mff.cuni.cz/_media/courses:mapreduce-tutorial:step-5-solution2.txt' -O 'step-5-solution2.pl'
rm -rf step-5-out-sol2; perl step-5-solution2.pl run /home/straka/wiki/cs-text-tiny/ step-5-out-sol2
less step-5-out-sol2/part-*