Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
|
courses:mapreduce-tutorial:step-13 [2012/01/25 23:00] straka |
courses:mapreduce-tutorial:step-13 [2012/01/31 15:54] (current) straka |
||
|---|---|---|---|
| Line 3: | Line 3: | ||
| You are given data consisting of (31-bit integer, string data) pairs. These are available in plain text format: | You are given data consisting of (31-bit integer, string data) pairs. These are available in plain text format: | ||
| ^ Path ^ Size ^ | ^ Path ^ Size ^ | ||
| - | | /home/straka/hadoop/example-inputs/ | + | | /net/projects/hadoop/examples/inputs/ |
| - | | /home/straka/hadoop/example-inputs/ | + | | /net/projects/hadoop/examples/inputs/ |
| - | | /home/straka/hadoop/example-inputs/ | + | | /net/projects/hadoop/examples/inputs/ |
| You can assume that the integers are uniformly distributed. | You can assume that the integers are uniformly distributed. | ||
| - | Your task is to sort these data. Your solution should work for TBs of data. For that reason, you must use multiple reducers. If your job is executed using //r// reducers, the output consists of //r// files, which when concatenated would produce sorted (key, value) pairs. In other words, each of the output files contains sorted (integer, data) pairs and all keys in one file are either smaller or larger than in other file. | + | Your task is to sort these data, comparing the key numerically and not lexicographically. The lines in the output must be the same as in the input, only in different order. |
| + | |||
| + | Your solution should work for TBs of data. For that reason, you must use multiple reducers. If your job is executed using //r// reducers, the output consists of //r// files, which when concatenated would produce sorted (key, value) pairs. In other words, each of the output files contains sorted (integer, data) pairs and all keys in one file are either smaller or larger than in other file. Your solution should work for any value //r// -- this value is given to [[.: | ||
| ===== Nonuniform data ===== | ===== Nonuniform data ===== | ||
| Line 15: | Line 17: | ||
| ^ Path ^ Size ^ | ^ Path ^ Size ^ | ||
| - | | /home/straka/hadoop/example-inputs/ | + | | /net/projects/hadoop/examples/inputs/ |
| - | | /home/straka/hadoop/example-inputs/ | + | | /net/projects/hadoop/examples/inputs/ |
| - | | /home/straka/hadoop/example-inputs/ | + | | /net/projects/hadoop/examples/inputs/ |
| Assume we want to produce //r// output files. One of the solutions is to perform two Hadoop jobs: | Assume we want to produce //r// output files. One of the solutions is to perform two Hadoop jobs: | ||
| Line 23: | Line 25: | ||
| - Find best //r-1// integer separators using the sampled data. | - Find best //r-1// integer separators using the sampled data. | ||
| - Run the second pass, using the separators to guide the partitioning. | - Run the second pass, using the separators to guide the partitioning. | ||
| + | |||
| + | |||
| + | ---- | ||
| + | |||
| + | < | ||
| + | <table style=" | ||
| + | <tr> | ||
| + | <td style=" | ||
| + | <td style=" | ||
| + | <td style=" | ||
| + | </tr> | ||
| + | </ | ||
| + | </ | ||