[ Skip to the content ]

Institute of Formal and Applied Linguistics Wiki

• ÚFAL Wiki Home
• ÚFAL Home
• Sitemap
• RSS
• ÚFAL calendar
• Recent changes
• Login
• Hledat:

[ Back to the navigation ]

Differences

This shows you the differences between two versions of the page.

View differences: Side by SideInline Go

Link to this comparison view

Both sides previous revision Previous revision 2010/03/25 10:01 stranak 2010/03/25 09:53 stranak dalsi OOV2010/03/24 18:11 stranak 2010/03/24 17:57 stranak 2010/03/24 17:56 stranak 2010/03/24 17:54 stranak pridany udaje o OOV pro Tides+dictfilt (Shabdanjali)2009/10/23 14:18 stranak 2009/10/23 14:17 stranak 2009/10/23 14:05 stranak 2009/10/23 14:05 stranak 2009/10/22 22:14 zeman After cleaning.2009/10/22 22:05 zeman Numbers of bytes to make the identification of the version of data more precise.2009/10/22 16:55 stranak 2009/10/22 16:54 stranak 2009/10/22 16:53 stranak 2009/10/22 15:58 stranak 2009/10/22 15:57 stranak 2009/10/22 15:36 stranak 2009/10/22 15:32 stranak 2009/10/22 14:22 zeman Tides statistics.2009/10/21 17:22 stranak 2009/10/21 14:35 stranak 2009/10/20 23:25 stranak 2009/10/20 23:24 stranak 2009/10/20 23:21 stranak 2009/10/20 23:08 stranak 2009/10/20 17:16 stranak 2009/10/16 10:09 zeman vytvořeno Go		2010/03/25 10:01 stranak 2010/03/25 09:53 stranak dalsi OOV2010/03/24 18:11 stranak 2010/03/24 17:57 stranak 2010/03/24 17:56 stranak 2010/03/24 17:54 stranak pridany udaje o OOV pro Tides+dictfilt (Shabdanjali)2009/10/23 14:18 stranak 2009/10/23 14:17 stranak 2009/10/23 14:05 stranak 2009/10/23 14:05 stranak 2009/10/22 22:14 zeman After cleaning.2009/10/22 22:05 zeman Numbers of bytes to make the identification of the version of data more precise.2009/10/22 16:55 stranak 2009/10/22 16:54 stranak 2009/10/22 16:53 stranak 2009/10/22 15:58 stranak 2009/10/22 15:57 stranak 2009/10/22 15:36 stranak 2009/10/22 15:32 stranak 2009/10/22 14:22 zeman Tides statistics.2009/10/21 17:22 stranak 2009/10/21 14:35 stranak 2009/10/20 23:25 stranak 2009/10/20 23:24 stranak 2009/10/20 23:21 stranak 2009/10/20 23:08 stranak 2009/10/20 17:16 stranak 2009/10/16 10:09 zeman vytvořeno Go Next revision Both sides next revision
pub-company:icon2009 [2009/10/22 16:55] stranak		pub-company:icon2009 [2009/10/22 22:05] zeman Numbers of bytes to make the identification of the version of data more precise.
Line 33:		Line 33:
	===== Data =====		===== Data =====
		+
	==== IIIT Tides ====		==== IIIT Tides ====
Line 38:		Line 39:
	A dataset originally collected for the DARPA-TIDES surprise-language contest in 2002, later refined at IIIT Hyderabad and provided for the NLP Tools Contest at ICON 2008. For availability, enquire at the IIIT (http://ltrc.iiit.ac.in/). If you have the data, make sure that you have the same number of sentences and tokens. The Hindi side has been provided in two encodings, WX romanization and UTF-8. Although they should be equivalent, we always work with UTF-8.		A dataset originally collected for the DARPA-TIDES surprise-language contest in 2002, later refined at IIIT Hyderabad and provided for the NLP Tools Contest at ICON 2008. For availability, enquire at the IIIT (http://ltrc.iiit.ac.in/). If you have the data, make sure that you have the same number of sentences and tokens. The Hindi side has been provided in two encodings, WX romanization and UTF-8. Although they should be equivalent, we always work with UTF-8.
-	| **Part** | **Sentences** | **Tokens en** | **Tokens hi** |	+	| **Part** | **Sentences** | **Tokens en** | **Bytes en** | **Tokens hi** | **Bytes hi** |
-	| train |  50,000 |  1,195,436 |  1,287,174 |	+	| train |  50,000 |  1,195,436 |  6,496,995 |  1,287,174 |  15,917,598 |
-	| dev |  1,000 |  21,842 |  23,851 |	+	| dev |  1,000 |  21,842 |  118,239 |  23,851 |  291,147 |
-	| test |  1,000 |  26,537 |  27,979 |	+	| test |  1,000 |  26,537 |  145,376 |  27,979 |  348,221 |
	The test data contain 1 reference translation per sentence.		The test data contain 1 reference translation per sentence.

---

[ Back to the navigation ] [ Back to the content ]