Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
courses:mapreduce-tutorial:step-8 [2012/01/29 21:12] straka |
courses:mapreduce-tutorial:step-8 [2012/01/31 15:55] (current) straka |
||
---|---|---|---|
Line 12: | Line 12: | ||
To use multiple reducers, the MR job must be executed by a cluster (even with one computer), not locally. The number of reducers is specified by '' | To use multiple reducers, the MR job must be executed by a cluster (even with one computer), not locally. The number of reducers is specified by '' | ||
- | perl script.pl | + | perl script.pl [-jt cluster_master | -c cluster_size [-w sec_to_wait]] [-r number_of_reducers] |
Optimal number of reducers is the same as the number of machines in the cluster, so that all the reducers can run in parallel at the same time. | Optimal number of reducers is the same as the number of machines in the cluster, so that all the reducers can run in parallel at the same time. | ||
Line 28: | Line 28: | ||
<code perl> | <code perl> | ||
- | package Partitioner; | + | package |
use Moose; | use Moose; | ||
with ' | with ' | ||
Line 39: | Line 39: | ||
... | ... | ||
- | package | + | package |
use Hadoop:: | use Hadoop:: | ||
my $runner = Hadoop:: | my $runner = Hadoop:: | ||
... | ... | ||
- | partitioner => Partitioner-> | + | partitioner => My::Partitioner-> |
...); | ...); | ||
... | ... | ||
Line 52: | Line 52: | ||
===== The order of keys during reduce ===== | ===== The order of keys during reduce ===== | ||
- | It is guaranteed that every reducer processes the keys in //ascending order//. | + | It is guaranteed that every reducer processes the keys in // |
On the other hand, the order of values belonging to one key is undefined. | On the other hand, the order of values belonging to one key is undefined. | ||
Line 60: | Line 60: | ||
Run one MR job on '/ | Run one MR job on '/ | ||
wget --no-check-certificate ' | wget --no-check-certificate ' | ||
- | rm -rf step-8-out-ex; | + | |
+ | # $EDITOR step-8-exercise.pl | ||
+ | | ||
less step-8-out-ex/ | less step-8-out-ex/ | ||
Line 66: | Line 68: | ||
You can also download the solution {{: | You can also download the solution {{: | ||
wget --no-check-certificate ' | wget --no-check-certificate ' | ||
- | rm -rf step-8-out-sol; | + | |
+ | # $EDITOR step-8-solution.pl | ||
+ | | ||
less step-8-out-sol/ | less step-8-out-sol/ | ||