[ Skip to the content ]

Institute of Formal and Applied Linguistics Wiki

• ÚFAL Wiki Home
• ÚFAL Home
• Sitemap
• RSS
• ÚFAL calendar
• Recent changes
• Login
• Hledat:

[ Back to the navigation ]

Differences

This shows you the differences between two versions of the page.

View differences: Side by SideInline Go

Link to this comparison view

Both sides previous revision Previous revision 2012/01/31 16:08 dusek 2012/01/28 23:14 majlis Added links to previous and next chapter.2012/01/26 23:18 straka 2012/01/26 23:18 straka 2012/01/26 23:17 straka 2012/01/26 23:16 straka 2012/01/26 23:15 straka 2012/01/26 23:14 straka 2012/01/25 23:37 straka 2012/01/25 23:36 straka 2012/01/25 23:35 straka 2012/01/25 23:30 straka 2012/01/25 23:15 straka 2012/01/25 22:19 straka 2012/01/25 15:46 straka vytvořeno Go		2012/01/31 16:08 dusek 2012/01/28 23:14 majlis Added links to previous and next chapter.2012/01/26 23:18 straka 2012/01/26 23:18 straka 2012/01/26 23:17 straka 2012/01/26 23:16 straka 2012/01/26 23:15 straka 2012/01/26 23:14 straka 2012/01/25 23:37 straka 2012/01/25 23:36 straka 2012/01/25 23:35 straka 2012/01/25 23:30 straka 2012/01/25 23:15 straka 2012/01/25 22:19 straka 2012/01/25 15:46 straka vytvořeno Go Next revision Both sides next revision
courses:mapreduce-tutorial:step-14 [2012/01/26 23:15] straka		courses:mapreduce-tutorial:step-14 [2012/01/26 23:16] straka
Line 9:		Line 9:
	Your model should contain all the unigrams, bigrams, ..., //N//-grams with the number of occurrences in the given corpus.		Your model should contain all the unigrams, bigrams, ..., //N//-grams with the number of occurrences in the given corpus.
-	As the size of the resulting corpus matters, you should represent the //N//-grams efficiently. Try using the following representation:	+	As the size of the resulting corpus matters, you should represent the //N//-grams efficiently. You can devise your own format, or you can use the following representation:
	* Compute the unique words of the corpus, filter out the words that have only one occurrence, sort them according to the number of their occurrences and number them from 1.		* Compute the unique words of the corpus, filter out the words that have only one occurrence, sort them according to the number of their occurrences and number them from 1.
	* In order to represent //N//-gram, use the //N// numbers of the words, followed by a 0. Store the numbers using variable-length encoding (smaller numbers take less bytes) -- use ''pack 'w*', @word_numbers, 0''.		* In order to represent //N//-gram, use the //N// numbers of the words, followed by a 0. Store the numbers using variable-length encoding (smaller numbers take less bytes) -- use ''pack 'w*', @word_numbers, 0''.

---

[ Back to the navigation ] [ Back to the content ]