[ Skip to the content ]

Institute of Formal and Applied Linguistics Wiki


[ Back to the navigation ]

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
courses:mapreduce-tutorial:step-4 [2012/01/24 21:10]
straka
courses:mapreduce-tutorial:step-4 [2012/01/31 09:40] (current)
straka Change Perl commandline syntax.
Line 1: Line 1:
 ====== MapReduce Tutorial : Counters ====== ====== MapReduce Tutorial : Counters ======
  
-Sometimes it is useful to +Sometimes it is useful to count events differently than outputting them as (key, value) pairs. For that reason Hadoop offers simple counter framework.
  
-{{.:step-3:exercise.txt|}} +Hadoop maintains a collection of pre-defined and user-defined counters. Every counter is identified by its group name and counter name. The group name and counter name is an arbitrary string -- for example group "Documents" and name "Number of words". To increment a counter, the following code can be used: 
-{{filelist>:snippets:dokuwiki:plugin-*.tar.gz&style=table&tableheader=1&tableshowdate=1&tableshowsize=1}}+ 
 +<code perl> 
 +sub map { 
 +  my ($self, $key, $value, $context) = @_; 
 + 
 +  $context->counter($group, $counter, $increment); 
 +
 +</code> 
 + 
 +At the end of computation, Hadoop prints an aggregated value of all the counters. 
 + 
 +===== Exercise ===== 
 + 
 +Run a Hadoop job on ''/home/straka/wiki/cs-text-medium'', which uses counters to count the number of articles according to their first letter (ignoring the case). You can download the template {{:courses:mapreduce-tutorial:step-4-exercise.txt|step-4-exercise.pl}}  and execute it. 
 +  wget --no-check-certificate 'https://wiki.ufal.ms.mff.cuni.cz/_media/courses:mapreduce-tutorial:step-4-exercise.txt' -O 'step-4-exercise.pl' 
 +  # NOW EDIT THE FILE 
 +  # $EDITOR step-4-exercise.pl 
 +  rm -rf step-4-out-ex; perl step-4-exercise.pl /home/straka/wiki/cs-text-medium/ step-4-out-ex 
 +  less step-4-out-ex/part-* 
 + 
 +==== Solution ==== 
 +You can also download the solution {{:courses:mapreduce-tutorial:step-4-solution.txt|step-4-solution.pl}} and check the correct output. 
 +  wget --no-check-certificate 'https://wiki.ufal.ms.mff.cuni.cz/_media/courses:mapreduce-tutorial:step-4-solution.txt' -O 'step-4-solution.pl' 
 +  # NOW VIEW THE FILE 
 +  # $EDITOR step-3-solution.pl 
 +  rm -rf step-4-out-sol; perl step-4-solution.pl /home/straka/wiki/cs-text-medium/ step-4-out-sol 
 +  less step-4-out-sol/part-* 
 + 
 +---- 
 + 
 +<html> 
 +<table style="width:100%"> 
 +<tr> 
 +<td style="text-align:left; width: 33%; "></html>[[step-3|Step 3]]: Basic mapper.<html></td> 
 +<td style="text-align:center; width: 33%; "></html>[[.|Overview]]<html></td> 
 +<td style="text-align:right; width: 33%; "></html>[[step-5|Step 5]]: Basic reducer.<html></td> 
 +</tr> 
 +</table> 
 +</html>

[ Back to the navigation ] [ Back to the content ]