Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
courses:mapreduce-tutorial:step-14 [2012/01/25 23:30] straka |
courses:mapreduce-tutorial:step-14 [2012/01/25 23:36] straka |
||
---|---|---|---|
Line 12: | Line 12: | ||
* Compute the unique words of the corpus, filter out the words that have only one occurrence, sort them according to the number of their occurrences and number them from 1. | * Compute the unique words of the corpus, filter out the words that have only one occurrence, sort them according to the number of their occurrences and number them from 1. | ||
* In order to represent //N//-gram, use the //N// numbers of the words, followed by 0. Store the numbers using variable-length encoding (smaller numbers take less bytes) -- use '' | * In order to represent //N//-gram, use the //N// numbers of the words, followed by 0. Store the numbers using variable-length encoding (smaller numbers take less bytes) -- use '' | ||
- | * One file of the resulting index should contain a sorted list of (n-gram representation, | + | * One file of the resulting index should contain a sorted list of (N-gram representation, |
- | * Every file should also be accompanied by the index -- the index contains every 1000-th //n-gram representation// | + | * Every data file should also be accompanied by the file index, which contains every 1000-th //N-gram representation// |
- | * If the resulting index should | + | * As in the sorting example, the //N-gram representation// |
+ | |||
+ | Try creating such index. Ideally, the sizes of resulting data files should be as equal as possible. | ||