[ Skip to the content ]

Institute of Formal and Applied Linguistics Wiki

• ÚFAL Wiki Home
• ÚFAL Home
• Sitemap
• RSS
• ÚFAL calendar
• Recent changes
• Login
• Hledat:

[ Back to the navigation ]

Differences

This shows you the differences between two versions of the page.

View differences: Side by SideInline Go

Link to this comparison view

Both sides previous revision Previous revision 2010/03/25 10:01 stranak 2010/03/25 09:53 stranak dalsi OOV2010/03/24 18:11 stranak 2010/03/24 17:57 stranak 2010/03/24 17:56 stranak 2010/03/24 17:54 stranak pridany udaje o OOV pro Tides+dictfilt (Shabdanjali)2009/10/23 14:18 stranak 2009/10/23 14:17 stranak 2009/10/23 14:05 stranak 2009/10/23 14:05 stranak 2009/10/22 22:14 zeman After cleaning.2009/10/22 22:05 zeman Numbers of bytes to make the identification of the version of data more precise.2009/10/22 16:55 stranak 2009/10/22 16:54 stranak 2009/10/22 16:53 stranak 2009/10/22 15:58 stranak 2009/10/22 15:57 stranak 2009/10/22 15:36 stranak 2009/10/22 15:32 stranak 2009/10/22 14:22 zeman Tides statistics.2009/10/21 17:22 stranak 2009/10/21 14:35 stranak 2009/10/20 23:25 stranak 2009/10/20 23:24 stranak 2009/10/20 23:21 stranak 2009/10/20 23:08 stranak 2009/10/20 17:16 stranak 2009/10/16 10:09 zeman vytvořeno Go		2010/03/25 10:01 stranak 2010/03/25 09:53 stranak dalsi OOV2010/03/24 18:11 stranak 2010/03/24 17:57 stranak 2010/03/24 17:56 stranak 2010/03/24 17:54 stranak pridany udaje o OOV pro Tides+dictfilt (Shabdanjali)2009/10/23 14:18 stranak 2009/10/23 14:17 stranak 2009/10/23 14:05 stranak 2009/10/23 14:05 stranak 2009/10/22 22:14 zeman After cleaning.2009/10/22 22:05 zeman Numbers of bytes to make the identification of the version of data more precise.2009/10/22 16:55 stranak 2009/10/22 16:54 stranak 2009/10/22 16:53 stranak 2009/10/22 15:58 stranak 2009/10/22 15:57 stranak 2009/10/22 15:36 stranak 2009/10/22 15:32 stranak 2009/10/22 14:22 zeman Tides statistics.2009/10/21 17:22 stranak 2009/10/21 14:35 stranak 2009/10/20 23:25 stranak 2009/10/20 23:24 stranak 2009/10/20 23:21 stranak 2009/10/20 23:08 stranak 2009/10/20 17:16 stranak 2009/10/16 10:09 zeman vytvořeno Go Next revision Both sides next revision
pub-company:icon2009 [2009/10/21 17:22] stranak		pub-company:icon2009 [2009/10/22 14:22] zeman Tides statistics.
Line 32:		Line 32:
	* Link to the PDF version of the paper; link to Biblio?		* Link to the PDF version of the paper; link to Biblio?
-	==== Out of Vocabulary ====	+	===== Data =====
		+
		+	==== IIIT Tides ====
		+
		+	A dataset originally collected for the DARPA-TIDES surprise-language contest in 2002, later refined at IIIT Hyderabad and provided for the NLP Tools Contest at ICON 2008. For availability, enquire at the IIIT (http://ltrc.iiit.ac.in/). If you have the data, make sure that you have the same number of sentences and tokens. The Hindi side has been provided in two encodings, WX romanization and UTF-8. Although they should be equivalent, we always work with UTF-8.
		+
		+	| **Part** | **Sentences** | **Tokens en** | **Tokens hi** |
		+	| train |  50,000 |  1,195,436 |  1,287,174 |
		+	| dev |  1,000 |  21,842 |  23,851 |
		+	| test |  1,000 |  26,537 |  27,979 |
		+
		+	The test data contain 1 reference translation per sentence.
		+
		+	===== Out of Vocabulary =====
	No data have been lemmatised, so  all the numbers mean forms.		No data have been lemmatised, so  all the numbers mean forms.
	^           Vocabulary Size               ^^^		^           Vocabulary Size               ^^^

---

[ Back to the navigation ] [ Back to the content ]