[ Skip to the content ]

Institute of Formal and Applied Linguistics Wiki

• ÚFAL Wiki Home
• ÚFAL Home
• Sitemap
• RSS
• ÚFAL calendar
• Recent changes
• Login
• Hledat:

[ Back to the navigation ]

Differences

This shows you the differences between two versions of the page.

View differences: Side by SideInline Go

Link to this comparison view

Both sides previous revision Previous revision 2012/01/31 16:08 dusek 2012/01/28 23:14 majlis Added links to previous and next chapter.2012/01/26 23:18 straka 2012/01/26 23:18 straka 2012/01/26 23:17 straka 2012/01/26 23:16 straka 2012/01/26 23:15 straka 2012/01/26 23:14 straka 2012/01/25 23:37 straka 2012/01/25 23:36 straka 2012/01/25 23:35 straka 2012/01/25 23:30 straka 2012/01/25 23:15 straka 2012/01/25 22:19 straka 2012/01/25 15:46 straka vytvořeno Go		2012/01/31 16:08 dusek 2012/01/28 23:14 majlis Added links to previous and next chapter.2012/01/26 23:18 straka 2012/01/26 23:18 straka 2012/01/26 23:17 straka 2012/01/26 23:16 straka 2012/01/26 23:15 straka 2012/01/26 23:14 straka 2012/01/25 23:37 straka 2012/01/25 23:36 straka 2012/01/25 23:35 straka 2012/01/25 23:30 straka 2012/01/25 23:15 straka 2012/01/25 22:19 straka 2012/01/25 15:46 straka vytvořeno Go Next revision Both sides next revision
courses:mapreduce-tutorial:step-14 [2012/01/25 23:15] straka		courses:mapreduce-tutorial:step-14 [2012/01/25 23:30] straka
Line 10:		Line 10:
	As the size of the resulting corpus matters, you should represent the //N//-grams efficiently. Try using the following representation:		As the size of the resulting corpus matters, you should represent the //N//-grams efficiently. Try using the following representation:
-	* Find the unique words of the corpus, sort them according to the number of their occurences	+	* Compute the unique words of the corpus, filter out the words that have only one occurrence, sort them according to the number of their occurrences and number them from 1.
		+	* In order to represent //N//-gram, use the //N// numbers of the words, followed by 0. Store the numbers using variable-length encoding (smaller numbers take less bytes) -- use ''pack 'w*', @word_numbers, 0''.
		+	* One file of the resulting index should contain a sorted list of (n-gram representation, occurrence), where //n-gram representation// is described above and //occurrence// is a variable-length encoded number of occurrences. No separators are necessary.
		+	* Every file should also be accompanied by the index -- the index contains every 1000-th //n-gram representation//
		+	* If the resulting index should consist of

---

[ Back to the navigation ] [ Back to the content ]