Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
courses:mapreduce-tutorial:step-8 [2012/01/25 22:09] straka |
courses:mapreduce-tutorial:step-8 [2012/01/29 21:04] straka |
||
---|---|---|---|
Line 1: | Line 1: | ||
====== MapReduce Tutorial : Multiple mappers, reducers and partitioning ====== | ====== MapReduce Tutorial : Multiple mappers, reducers and partitioning ====== | ||
- | In order to achieve parallelism, mappers and reducers | + | A Hadoop job, which is expected |
===== Multiple mappers ===== | ===== Multiple mappers ===== | ||
Line 13: | Line 13: | ||
To use multiple reducers, the MR job must be executed by a cluster (even with one computer), not locally. The number of reducers is specified by '' | To use multiple reducers, the MR job must be executed by a cluster (even with one computer), not locally. The number of reducers is specified by '' | ||
perl script.pl run [-jt cluster_master | -c cluster_size [-w sec_to_wait]] [-r number_of_reducers] | perl script.pl run [-jt cluster_master | -c cluster_size [-w sec_to_wait]] [-r number_of_reducers] | ||
+ | |||
+ | Optimal number of reducers is the same as the number of machines in the cluster, so that all the reducers can run in parallel at the same time. | ||
==== Partitioning ==== | ==== Partitioning ==== | ||
Line 50: | Line 52: | ||
On the other hand, the order of values belonging to one key is undefined. | On the other hand, the order of values belonging to one key is undefined. | ||
- | ===== Example | + | ===== Exercise |
+ | |||
+ | Run one MR job on '/ | ||
+ | wget --no-check-certificate ' | ||
+ | rm -rf step-8-out-ex; | ||
+ | less step-8-out-ex/ | ||
+ | |||
+ | ==== Solution ==== | ||
+ | You can also download the solution {{: | ||
+ | wget --no-check-certificate ' | ||
+ | rm -rf step-8-out-sol; | ||
+ | less step-8-out-sol/ | ||
- | Run a MR job on '/ | ||
- | {{: | + | ---- |
+ | < | ||
+ | <table style=" | ||
+ | <tr> | ||
+ | <td style=" | ||
+ | <td style=" | ||
+ | <td style=" | ||
+ | </tr> | ||
+ | </ | ||
+ | </ |