[ Skip to the content ]

Institute of Formal and Applied Linguistics Wiki

[ Back to the navigation ]

You are here: start » user » zeman » treebanks » te

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
Next revision Both sides next revision
user:zeman:treebanks:te [2012/03/22 11:34]
zeman Training data size (both sentences and words) was identical in ICON 2009 and 2010. 		user:zeman:treebanks:te [2012/03/22 16:47]
zeman ICON 2010 Telugu data size.
Line 43: Line 43:
 ==== Size ==== ==== Size ====
  
-HyDT-Telugu shows dependencies between chunks, not words. The node/tree ratio is thus much lower than in other treebanks. The ICON 2009 version came with a data split into three parts: training, development and test; the same data was also distributed for ICON 2010:+HyDT-Telugu shows dependencies between chunks, not words. The node/tree ratio is thus much lower than in other treebanks. The ICON 2009 version came with a data split into three parts: training, development and test:
  
-^ Part ^ Sentences ^ Chunks ^ Ratio ^ Words ^ Ratio ^ +^ Part ^ Sentences ^ Chunks ^ Ratio ^ 
-| Training | 1400 | ? | ? | 7602 | 5.43 | +| Training    |  1456 |  5494 | 3.77 | 
-| Development | 150 | ? | ? | 839 | 5.59 | +| Development |   150 |   675 | 4.50 | 
-| Test | 150 | ? | ? | 836 | 5.57 | +| Test        |   150 |   583 | 3.89 | 
-| TOTAL | 1700 | ? | ? | 9277 | 5.46 |+| TOTAL       |  1756 |  6752 | 3.85 |
  
-We drew our training and test data from the ICON 2010 datasets but we have fewer sentences – why?+As for ICON 2010, the data description in [[http://ltrc.iiit.ac.in/nlptools2010/files/documents/toolscontest10-workshoppaper-final.pdf|(Husain et al., 2010)]] does not match the data that we downloaded during the contest. They indicate the number of words, while we can count the number of nodes, i.e. chunks. Anyway, the number of training sentences should match and it does not. Also note that they give the same number of words as they gave for ICON 2009 in [[http://ltrc.iiit.ac.in/nlptools2009/CR/intro-husain.pdf|(Husain et al., 2009)]]. In any case, the training data shrank during the year (clean up?) In the following table, we give both the published and real numbers of sentences, the published number of words and the counted number of chunks (nodes).
  
-^ Part ^ Sentences ^ Chunks ^ Ratio ^ +^ Part ^ Sentences ^ Chunks ^ Ratio ^ PSentences ^ Words ^ Ratio ^ 
-| Training |  1300 |  5125 | 3.94 | +| Training    |  1300 |  5125 |  3.94 |  1400 |  7602 |  5.43 | 
-| Test     |   150 |   597 | 3.98 | +| Development |   150 |   597 |  3.98 |   150 |   839 |  5.59 | 
-| TOTAL    |  1450 |  5722 | 3.95 |+| Test        |   150 |   599 |  3.99 |   150 |   836 |  5.57 | 
 +| TOTAL       |  1600 |  6321 |  3.95 |  1700 |  9277 |  5.46 |
  
 ==== Inside ==== ==== Inside ====

[ Back to the navigation ] [ Back to the content ]