| Both sides previous revision
Previous revision
Next revision
|
Previous revision
Next revision
Both sides next revision
|
user:zeman:treebanks:te [2012/03/22 11:46]
zeman ICON 2009 Telugu data size. |
user:zeman:treebanks:te [2012/03/22 17:06]
zeman Inside. |
| | TOTAL | 1756 | 6752 | 3.85 | | | TOTAL | 1756 | 6752 | 3.85 | |
| |
| The data distributed for ICON 2010 was slightly smaller, maybe it had been cleaned up? Note that the number of training words, 7602, is identical to the number published for ICON 2009. I cannot verify it because I only see chunks, not words in the CoNLL data format. | As for ICON 2010, the data description in [[http://ltrc.iiit.ac.in/nlptools2010/files/documents/toolscontest10-workshoppaper-final.pdf|(Husain et al., 2010)]] does not match the data that we downloaded during the contest. They indicate the number of words, while we can count the number of nodes, i.e. chunks. Anyway, the number of training sentences should match and it does not. Also note that they give the same number of words as they gave for ICON 2009 in [[http://ltrc.iiit.ac.in/nlptools2009/CR/intro-husain.pdf|(Husain et al., 2009)]]. In any case, the training data shrank during the year (clean up?) In the following table, we give both the published and real numbers of sentences, the published number of words and the counted number of chunks (nodes). |
| |
| ^ Part ^ Sentences ^ Chunks ^ Ratio ^ Words ^ Ratio ^ | ^ Part ^ Sentences ^ Chunks ^ Ratio ^ PSentences ^ Words ^ Ratio ^ |
| | Training | 1400 | ? | ? | 7602 | 5.43 | | | Training | 1300 | 5125 | 3.94 | 1400 | 7602 | 5.43 | |
| | Development | 150 | ? | ? | 839 | 5.59 | | | Development | 150 | 597 | 3.98 | 150 | 839 | 5.59 | |
| | Test | 150 | ? | ? | 836 | 5.57 | | | Test | 150 | 599 | 3.99 | 150 | 836 | 5.57 | |
| | TOTAL | 1700 | ? | ? | 9277 | 5.46 | | | TOTAL | 1600 | 6321 | 3.95 | 1700 | 9277 | 5.46 | |
| | |
| We drew our training and test data from the ICON 2010 datasets but we have fewer sentences – why? | |
| | |
| ^ Part ^ Sentences ^ Chunks ^ Ratio ^ | |
| | Training | 1300 | 5125 | 3.94 | | |
| | Test | 150 | 597 | 3.98 | | |
| | TOTAL | 1450 | 5722 | 3.95 | | |
| |
| ==== Inside ==== | ==== Inside ==== |
| |
| The text uses the [[http://ltrc.iiit.ac.in/nlptools2010/files/documents/map.pdf|WX encoding]] of Indian letters. If we know what the original script is (Bengali in this case) we can map the WX encoding to the original characters in UTF-8. WX uses English letters so if there was embedded English (or other string using Latin letters) it will probably get lost during the conversion. | The text uses the [[http://ltrc.iiit.ac.in/nlptools2010/files/documents/map.pdf|WX encoding]] of Indian letters. If we know what the original script is (Telugu in this case) we can map the WX encoding to the original characters in UTF-8. WX uses English letters so if there was embedded English (or other string using Latin letters) it will probably get lost during the conversion. |
| |
| The CoNLL format contains only the chunk heads. The native SSF format shows the other words in the chunk, too, but it does not capture intra-chunk dependency relations. This is an example of a multi-word chunk: | The CoNLL format contains only the chunk heads. The native SSF format shows the other words in the chunk, too, but it does not capture intra-chunk dependency relations. This is an example of a multi-word chunk: |
| |
| <code>3 (( NP <fs af='rumAla,n,,sg,,d,0,0' head="rumAla" drel=k2:VGF name=NP3> | <code>3 (( NP <fs af='AdavAlYlu,n,,sg,,,0,0_e' head='AdavAlYle' pbank='ARG3' name='NP3'> |
| 3.1 ekatA QC <fs af='eka,num,,,,,,'> | 3.1 932 QC <fs af='932,num,,,,,,'> |
| 3.2 ledisa JJ <fs af='ledisa,unk,,,,,,'> | 3.2 maMxi CL <fs af='maMxi,n,,pl,,d,0,0'> |
| 3.3 rumAla NN <fs af='rumAla,n,,sg,,d,0,0' name="rumAla"> | 3.3 AdavAlYle NN <fs af='AdavAlYlu,n,,sg,,,0,0_e' name='AdavAlYle'> |
| ))</code> | ))</code> |
| |
| According to [[http://ltrc.iiit.ac.in/nlptools2010/files/documents/toolscontest10-workshoppaper-final.pdf|(Husain et al., 2010)]], in the ICON 2010 version, the chunk tags, POS tags and inter-chunk dependencies (topology + tags) were annotated manually. The rest (lemma, morphosyntactic features, headword of chunk) was marked automatically. | According to [[http://ltrc.iiit.ac.in/nlptools2010/files/documents/toolscontest10-workshoppaper-final.pdf|(Husain et al., 2010)]], in the ICON 2010 version, the chunk tags, POS tags and inter-chunk dependencies (topology + tags) were annotated manually. The rest (lemma, morphosyntactic features, headword of chunk) was marked automatically. |
| |
| Note: There have been cycles in the Hindi part of HyDT but no such problem occurs in the Bengali part. | Note: There have been cycles in the Hindi part of HyDT but no such problem occurs in the Telugu part. |
| |
| ==== Sample ==== | ==== Sample ==== |
