[ Skip to the content ]

Institute of Formal and Applied Linguistics Wiki

• ÚFAL Wiki Home
• ÚFAL Home
• Sitemap
• RSS
• ÚFAL calendar
• Recent changes
• Login
• Hledat:

[ Back to the navigation ]

Differences

This shows you the differences between two versions of the page.

View differences: Side by SideInline Go

Link to this comparison view

Both sides previous revision Previous revision 2010/03/25 10:01 stranak 2010/03/25 09:53 stranak dalsi OOV2010/03/24 18:11 stranak 2010/03/24 17:57 stranak 2010/03/24 17:56 stranak 2010/03/24 17:54 stranak pridany udaje o OOV pro Tides+dictfilt (Shabdanjali)2009/10/23 14:18 stranak 2009/10/23 14:17 stranak 2009/10/23 14:05 stranak 2009/10/23 14:05 stranak 2009/10/22 22:14 zeman After cleaning.2009/10/22 22:05 zeman Numbers of bytes to make the identification of the version of data more precise.2009/10/22 16:55 stranak 2009/10/22 16:54 stranak 2009/10/22 16:53 stranak 2009/10/22 15:58 stranak 2009/10/22 15:57 stranak 2009/10/22 15:36 stranak 2009/10/22 15:32 stranak 2009/10/22 14:22 zeman Tides statistics.2009/10/21 17:22 stranak 2009/10/21 14:35 stranak 2009/10/20 23:25 stranak 2009/10/20 23:24 stranak 2009/10/20 23:21 stranak 2009/10/20 23:08 stranak 2009/10/20 17:16 stranak 2009/10/16 10:09 zeman vytvořeno Go Next revision		Previous revision 2010/03/25 10:01 stranak 2010/03/25 09:53 stranak dalsi OOV2010/03/24 18:11 stranak 2010/03/24 17:57 stranak 2010/03/24 17:56 stranak 2010/03/24 17:54 stranak pridany udaje o OOV pro Tides+dictfilt (Shabdanjali)2009/10/23 14:18 stranak 2009/10/23 14:17 stranak 2009/10/23 14:05 stranak 2009/10/23 14:05 stranak 2009/10/22 22:14 zeman After cleaning.2009/10/22 22:05 zeman Numbers of bytes to make the identification of the version of data more precise.2009/10/22 16:55 stranak 2009/10/22 16:54 stranak 2009/10/22 16:53 stranak 2009/10/22 15:58 stranak 2009/10/22 15:57 stranak 2009/10/22 15:36 stranak 2009/10/22 15:32 stranak 2009/10/22 14:22 zeman Tides statistics.2009/10/21 17:22 stranak 2009/10/21 14:35 stranak 2009/10/20 23:25 stranak 2009/10/20 23:24 stranak 2009/10/20 23:21 stranak 2009/10/20 23:08 stranak 2009/10/20 17:16 stranak 2009/10/16 10:09 zeman vytvořeno Go Next revision Both sides next revision
pub-company:icon2009 [2009/10/21 17:22] stranak		pub-company:icon2009 [2009/10/22 15:36] stranak
Line 32:		Line 32:
	* Link to the PDF version of the paper; link to Biblio?		* Link to the PDF version of the paper; link to Biblio?
-	==== Out of Vocabulary ====	+	===== Data =====
		+
		+	==== IIIT Tides ====
		+
		+	A dataset originally collected for the DARPA-TIDES surprise-language contest in 2002, later refined at IIIT Hyderabad and provided for the NLP Tools Contest at ICON 2008. For availability, enquire at the IIIT (http://ltrc.iiit.ac.in/). If you have the data, make sure that you have the same number of sentences and tokens. The Hindi side has been provided in two encodings, WX romanization and UTF-8. Although they should be equivalent, we always work with UTF-8.
		+
		+	| **Part** | **Sentences** | **Tokens en** | **Tokens hi** |
		+	| train |  50,000 |  1,195,436 |  1,287,174 |
		+	| dev |  1,000 |  21,842 |  23,851 |
		+	| test |  1,000 |  26,537 |  27,979 |
		+
		+	The test data contain 1 reference translation per sentence.
		+
		+	===== Out of Vocabulary =====
	No data have been lemmatised, so  all the numbers mean forms.		No data have been lemmatised, so  all the numbers mean forms.
	^           Vocabulary Size               ^^^		^           Vocabulary Size               ^^^
Line 47:		Line 60:
	^         Coverage               ^^^^^^^^^		^         Coverage               ^^^^^^^^^
-	|                   | **tokens seen in train**  ||||  **types seen in train**  ||||	+	|                   | **tokens unseen in train**  ||  **types unseen in train**  ||
-	|                   |  //Tides//  ||  //Tides+DP//  ||  //Tides//  || //Tides+DP// ||	+	|                   |  //Tides//  |  //Tides+DP//  |  //Tides//  | //Tides+DP//   |
-	|                   | abs |  OOV  | abs |    OOV   | abs |  OOV  | abs |    OOV   |	+	| **Tides-test-en** |   369       |     348        |   363       |       343      |
-	| **Tides-test-en** |             |                |             |                |	+	| **Tides-test-hi** |   839       |     830        |   642       |       633      |
-	| **Tides-test-hi** |             |                |             |                |	+	| **Tides-dev-en**  |   464       |     421        |   459       |       418      |
-	| **Tides-dev-en**  |             |                |             |                |	+	| **Tides-dev-hi**  |   619       |     607        |   580       |       568      |
-	| **Tides-dev-hi**  |             |                |             |                |	+

---

[ Back to the navigation ] [ Back to the content ]