Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
user:zeman:treebanks [2011/11/10 13:14] zeman Catalan 2007 sample. |
user:zeman:treebanks [2012/01/17 10:41] zeman Swedish. |
||
---|---|---|---|
Line 1: | Line 1: | ||
====== Treebanks for Various Languages ====== | ====== Treebanks for Various Languages ====== | ||
- | ===== Arabic (ar) ===== | + | * [[user:zeman:treebanks: |
- | + | * [[user:zeman: | |
- | [[http:// | + | * [[user:zeman:treebanks:bn|Bengali (bn)]] |
- | + | * [[user:zeman:treebanks:ca|Catalan | |
- | ==== Versions ==== | + | |
- | + | | |
- | * Original PADT 1.0 as distributed by the LDC | + | |
- | * CoNLL 2006 | + | |
- | * CoNLL 2007 | + | * [[user: |
- | + | * [[user:zeman: | |
- | The CoNLL 2007 version reportedly improves over CoNLL 2006 in quality of morphological annotation. Both CoNLL versions miss important parts of the original PADT annotation (see below). | + | * [[user:zeman:treebanks:et|Estonian (et)]] |
- | + | * [[user:zeman:treebanks:eu|Basque | |
- | ==== Obtaining and License ==== | + | |
- | + | * [[user:zeman:treebanks: | |
- | The original PADT 1.0 is distributed by the LDC under the catalogue number | + | * [[user:zeman:treebanks:hi|Hindi (hi)]] |
- | + | * [[user:zeman:treebanks:hu|Hungarian (hu)]] | |
- | * non-commercial research usage | + | * [[user:zeman:treebanks:it|Italian |
- | * no redistribution | + | |
- | * cite [[http:// | + | |
- | + | * [[user:zeman:treebanks: | |
- | The CoNLL 2006 and 2007 versions are obtainable upon request under similar license terms. Their publication in the LDC together with the other CoNLL treebanks | + | * [[user:zeman:treebanks:pt|Portuguese |
- | + | * [[user:zeman:treebanks: | |
- | PADT was created by members of the [[http:// | + | * [[user:zeman:treebanks:ru|Russian |
- | + | * [[user:zeman:treebanks:sl|Slovene (sl)]] | |
- | ==== Domain ==== | + | * [[user:zeman:treebanks:sv|Swedish |
- | + | ||
- | Newswire text from press agencies (Agence France Presse, Ummah, Al Hayat, An Nahar, Xinhua 2001-2003). | + | |
- | + | ||
- | ==== Size ==== | + | |
- | + | ||
- | According to their website, the original PADT 1.0 contains 113,500 tokens annotated analytically. The CoNLL 2006 version contains 59752 tokens in 1606 sentences, yielding 37.21 tokens per sentence on average (CoNLL 2006 data split: 54379 tokens / 1460 sentences training, 5373 tokens / 146 sentences test). The CoNLL 2007 version contains 116,793 tokens in 3043 sentences, yielding 38.38 tokens per sentence on average (CoNLL 2007 data split: 111,669 tokens / 2912 sentences training, 5124 tokens / 131 sentences test). | + | |
- | + | ||
- | As noted in (Nivre et al., 2007), “the parsing units in this treebank are in many cases larger than conventional sentences, which partly explains the high average number of tokens per sentence.” | + | |
- | + | ||
- | ==== References ==== | + | |
- | + | ||
- | * Website | + | |
- | * http:// | + | |
- | * Data | + | |
- | * Jan Hajič, Otakar Smrž, Petr Zemánek, Petr Pajas, Jan Šnaidauf, Emanuel Beška, Jakub Kráčmar, Kamila Hassanová: //Prague Arabic Dependency Treebank 1.0// (LDC2004T23). Linguistic Data Consortium, Philadelphia, | + | |
- | * Principal publications | + | |
- | * Jan Hajič, Otakar Smrž, Petr Zemánek, Jan Šnaidauf, Emanuel Beška: | + | |
- | + | ||
- | ==== Inside ==== | + | |
- | + | ||
- | The original PADT 1.0 is distributed in the [[: | + | |
- | + | ||
- | Word forms and lemmas are vocalized, i.e. they contain diacritics for short vowels as well as consonant gemination and a few other things. The CoNLL 2006 version includes [[http:// | + | |
- | + | ||
- | Note that tokenization of Arabic typically includes splitting original words (inserting spaces between letters), not just separating punctuation from words. Example: وبالفالوجة = wabiālfālūjah = wa/CONJ + bi/PREP + AlfAlwjp/ | + | |
- | + | ||
- | The original PADT 1.0 uses 10-character positional morphological tags whose documentation is hard to find. The CoNLL 2006 version converts the tags to the three CoNLL columns, CPOS, POS and FEAT, most of the information being encoded as pipe-separated attribute-value assignments in FEAT. There //should// be a 1-1 mapping between the PADT positional tags and the CoNLL 2006 annotation. The CoNLL 2007 version uses a tag conversion different from CoNLL 2006. Both CoNLL distributions contain a README file with a brief description of the parts of speech and features. Use [[http:// | + | |
- | + | ||
- | The guidelines for syntactic annotation are documented in the [[http:// | + | |
- | + | ||
- | ==== Sample ==== | + | |
- | + | ||
- | The first two sentences of the CoNLL 2006 training data: | + | |
- | + | ||
- | | 1 | غِيابُ_giyAbu | غِياب_giyAb | N | N | case=1< | + | |
- | | 2 | فُؤاد_fu& | + | |
- | | 3 | كَنْعان_kanoEAn | كَنْعان_kanoEAn | Z | Z | _ | 1 | Atr | _ | _ | | + | |
- | | |||||||||| | + | |
- | | 1 | فُؤاد_fu& | + | |
- | | 2 | كَنْعان_kanoEAn | كَنْعان_kanoEAn | Z | Z | _ | 9 | Sb | _ | _ | | + | |
- | | 3 | ،_, | ،_, | G | G | _ | 2 | AuxG | _ | _ | | + | |
- | | 4 | رائِد_rA}id | رائِد_rA}id | N | N | _ | 2 | Atr | _ | _ | | + | |
- | | 5 | القِصَّة_AlqiS~ap | قِصَّة_qiS~ap | N | N | gen=F< | + | |
- | | 6 | القَصِيرَةِ_AlqaSiyrapi | قَصِير_qaSiyr | A | A | gen=F< | + | |
- | | 7 | فِي_fiy | فِي_fiy | P | P | _ | 4 | AuxP | _ | _ | | + | |
- | | 8 | لُبْنانِ_lubonAni | لُبْنان_lubonAn | Z | Z | case=2< | + | |
- | | 9 | رَحَلَ_raHala | رَحَل-َ_raHal-a | V | VP | pers=3< | + | |
- | | 10 | مَساءَ_masA' | + | |
- | | 11 | أَمْسِ_> | + | |
- | | 12 | عَن_Ean | عَن_Ean | P | P | _ | 9 | AuxP | _ | _ | | + | |
- | | 13 | 81_81 | 81_81 | Q | Q | _ | 12 | Adv | _ | _ | | + | |
- | | 14 | عاماً_EAmAF | عام_EAm | N | N | gen=M< | + | |
- | | 15 | ._. | ._. | G | G | _ | 0 | AuxK | _ | _ | | + | |
- | + | ||
- | The first sentence of the CoNLL 2006 test data: | + | |
- | + | ||
- | | 1 | اِتِّفاقٌ_Ait~ifAqN | اِتِّفاق_Ait~ifAq | N | N | case=1< | + | |
- | | 2 | بَيْنَ_bayona | بَيْنَ_bayona | P | P | _ | 1 | AuxP | _ | _ | | + | |
- | | 3 | لُبْنانِ_lubonAni | لُبْنان_lubonAn | Z | Z | case=2< | + | |
- | | 4 | وَ_wa | وَ_wa | C | C | _ | 2 | Coord | _ | _ | | + | |
- | | 5 | سُورِيَّةٍ_suwriy~apK | سُورِيا_suwriyA | Z | Z | gen=F< | + | |
- | | 6 | عَلَى_EalaY | عَلَى_EalaY | P | P | _ | 1 | AuxP | _ | _ | | + | |
- | | 7 | رَفْعِ_rafoEi | رَفْع_rafoE | N | N | case=2< | + | |
- | | 8 | مُسْتَوَى_musotawaY | مُسْتَوَى_musotawaY | N | N | _ | 7 | Atr | _ | _ | | + | |
- | | 9 | التَبادُلِ_AltabAduli | تَبادُل_tabAdul | N | N | case=2< | + | |
- | | 10 | التِجارِيِّ_AltijAriy~i | تِجارِيّ_tijAriy~ | A | A | case=2< | + | |
- | | 11 | إِلَى_< | + | |
- | | 12 | 500_500 | 500_500 | Q | Q | _ | 11 | Atr | _ | _ | | + | |
- | | 13 | مِلْيُونِ_miloyuwni | مِلْيُون_miloyuwn | N | N | case=2< | + | |
- | | 14 | دُولارٍ_duwlArK | دُولار_duwlAr | N | N | case=2< | + | |
- | + | ||
- | The first sentence of the CoNLL 2007 training data: | + | |
- | + | ||
- | | 1 | تَعْدادُ | تَعْداد_1 | N | N- | Case=1< | + | |
- | | 2 | سُكّانِ | ساكِن_1 | N | N- | Case=2< | + | |
- | | 3 | 22 | [DEFAULT] | Q | Q- | _ | 2 | Atr | _ | _ | | + | |
- | | 4 | دَوْلَةً | دَوْلَة_1 | N | N- | Gender=F< | + | |
- | | 5 | عَرَبِيَّةً | عَرَبِيّ_1 | A | A- | Gender=F< | + | |
- | | 6 | سَ | سَ_FUT | F | F- | _ | 7 | AuxM | _ | _ | | + | |
- | | 7 | يَرْتَفِعُ | اِرْتَفَع_1 | V | VI | Mood=I< | + | |
- | | 8 | إِلَى | إِلَى_1 | P | P- | _ | 7 | AuxP | _ | _ | | + | |
- | | 9 | 654 | [DEFAULT] | Q | Q- | _ | 8 | Adv | _ | _ | | + | |
- | | 10 | مِلْيُونَ | مِلْيُون_1 | N | N- | Case=4< | + | |
- | | 11 | نَسَمَةٍ | نَسَمَة_1 | N | N- | Gender=F< | + | |
- | | 12 | فِي | فِي_1 | P | P- | _ | 7 | AuxP | _ | _ | | + | |
- | | 13 | مُنْتَصَفِ | مُنْتَصَف_1 | N | N- | Case=2< | + | |
- | | 14 | القَرْنِ | قَرْن_1 | N | N- | Case=2< | + | |
- | + | ||
- | The first sentence of the CoNLL 2007 test data: | + | |
- | + | ||
- | | 1 | مُقاوَمَةُ | مُقاوَمَة_1 | N | N- | Gender=F< | + | |
- | | 2 | زَواجِ | زَواج_1 | N | N- | Case=2< | + | |
- | | 3 | الطُلّابِ | طالِب_1 | N | N- | Case=2< | + | |
- | | 4 | العُرْفِيِّ | عُرْفِيّ_1 | A | A- | Case=2< | + | |
- | + | ||
- | ==== Parsing ==== | + | |
- | + | ||
- | Nonprojectivities in PADT are rare. Only 431 of the 116,793 tokens in the CoNLL 2007 version are attached nonprojectively (0.37%). | + | |
- | + | ||
- | The results of the CoNLL 2006 shared task are [[http:// | + | |
- | + | ||
- | ^ Parser (Authors) ^ LAS ^ UAS ^ | + | |
- | | MST (McDonald et al.) | 66.91 | 79.34 | | + | |
- | | Basis (O' | + | |
- | | Malt (Nivre et al.) | 66.71 | 77.52 | | + | |
- | | Edinburgh (Riedel et al.) | 66.65 | 78.62 | | + | |
- | + | ||
- | The results of the CoNLL 2007 shared task are [[http:// | + | |
- | + | ||
- | ^ Parser (Authors) ^ LAS ^ UAS ^ | + | |
- | | Malt (Nilsson et al.) | 76.52 | 85.81 | | + | |
- | | Nakagawa | 75.08 | 86.09 | | + | |
- | | Malt (Hall et al.) | 74.75 | 84.21 | | + | |
- | | Sagae | 74.71 | 84.04 | | + | |
- | | Chen | 74.65 | 83.49 | | + | |
- | | Titov et al. | 74.12 | 83.18 | | + | |
- | + | ||
- | The two Malt parser results of 2007 (single malt and blended) are described in [[http:// | + | |
- | + | ||
- | ===== Bulgarian | + | |
- | + | ||
- | [[http:// | + | |
- | + | ||
- | ==== Versions ==== | + | |
- | + | ||
- | * Original BTB in native format | + | |
- | * CoNLL 2006 (BulTreeBank-DP) | + | |
- | + | ||
- | The original BTB is based on HPSG (head-driven phrase-structure grammar). The CoNLL version contains only the dependency information encoded in HPSG BulTreeBank. | + | |
- | + | ||
- | ==== Obtaining and License ==== | + | |
- | + | ||
- | Only the CoNLL version seems to be distributed but you may ask the creators about the HPSG version. For the dependency version, print the [[http:// | + | |
- | + | ||
- | * research usage | + | |
- | * no redistribution | + | |
- | * cite [[http:// | + | |
- | + | ||
- | BTB was created by members of the [[http:// | + | |
- | + | ||
- | ==== Domain ==== | + | |
- | + | ||
- | Unknown (“A set of Bulgarian sentences marked-up with detailed syntactic information. These sentences are mainly extracted from authentic Bulgarian texts. They are chosen with regards two criteria. First, they cover the variety of syntactic structures of Bulgarian. Second, they show the statistical distribution of these phenomena in real texts.”) At least part of it is probably news (Novinar, Sega, Standart). | + | |
- | + | ||
- | ==== Size ==== | + | |
- | + | ||
- | The CoNLL 2006 version contains 196,151 tokens in 13221 sentences, yielding 14.84 tokens per sentence on average (CoNLL 2006 data split: 190,217 tokens / 12823 sentences training, 5934 tokens / 398 sentences test). | + | |
- | + | ||
- | ==== References ==== | + | |
- | + | ||
- | * Website | + | |
- | * http:// | + | |
- | * Data | + | |
- | * //no separate citation// | + | |
- | * Principal publications | + | |
- | * Kiril Simov, Petya Osenova, Alexander Simov, Milen Kouylekov: //Design and Implementation of the Bulgarian HPSG-based Treebank.// In: Erhard Hinrichs, Kiril Simov (eds.): Journal of Research on Language and Computation, | + | |
- | * Documentation | + | |
- | * Kiril Simov, Petya Osenova, Milena Slavcheva: | + | |
- | * Petya Osenova, Kiril Simov: [[http:// | + | |
- | * http:// | + | |
- | + | ||
- | ==== Inside ==== | + | |
- | + | ||
- | The original morphosyntactic tags have been converted to fit into the three columns (CPOS, POS and FEAT) columns of the CoNLL format. There //should// be a 1-1 mapping between the [[http:// | + | |
- | + | ||
- | The morphological analysis does not include lemmas. The morphosyntactic tags have been assigned | + | |
- | + | ||
- | The guidelines for syntactic annotation are documented in the other [[http:// | + | |
- | + | ||
- | ==== Sample ==== | + | |
- | + | ||
- | The first three sentences of the CoNLL 2006 training data: | + | |
- | + | ||
- | | 1 | Глава | _ | N | Nc | _ | 0 | ROOT | 0 | ROOT | | + | |
- | | 2 | трета | _ | M | Mo | gen=f< | + | |
- | | |||||||||| | + | |
- | | 1 | НАРОДНО | _ | A | An | gen=n< | + | |
- | | 2 | СЪБРАНИЕ | _ | N | Nc | gen=n< | + | |
- | | |||||||||| | + | |
- | | 1 | Народното | _ | A | An | gen=n< | + | |
- | | 2 | събрание | _ | N | Nc | gen=n< | + | |
- | | 3 | осъществява | _ | V | Vpi | trans=t< | + | |
- | | 4 | законодателната | _ | A | Af | gen=f< | + | |
- | | 5 | власт | _ | N | Nc | _ | 3 | obj | 3 | obj | | + | |
- | | 6 | и | _ | C | Cp | _ | 3 | conj | 3 | conj | | + | |
- | | 7 | упражнява | _ | V | Vpi | trans=t< | + | |
- | | 8 | парламентарен | _ | A | Am | gen=m< | + | |
- | | 9 | контрол | _ | N | Nc | gen=m< | + | |
- | | 10 | . | _ | Punct | Punct | _ | 3 | punct | 3 | punct | | + | |
- | + | ||
- | The first three sentences of the CoNLL 2006 test data: | + | |
- | + | ||
- | | 1 | Единственото | _ | A | An | gen=n< | + | |
- | | 2 | решение | _ | N | Nc | gen=n< | + | |
- | | |||||||||| | + | |
- | | 1 | Ерик | _ | N | Np | gen=m< | + | |
- | | 2 | Франк | _ | N | Np | gen=m< | + | |
- | | 3 | Ръсел | _ | H | Hm | gen=m< | + | |
- | | |||||||||| | + | |
- | | 1 | Пълен | _ | A | Am | gen=m< | + | |
- | | 2 | мрак | _ | N | Nc | gen=m< | + | |
- | | 3 | и | _ | C | Cp | _ | 2 | conj | 2 | conj | | + | |
- | | 4 | пълна | _ | A | Af | gen=f< | + | |
- | | 5 | самота | _ | N | Nc | _ | 2 | conjarg | 2 | conjarg | | + | |
- | | 6 | . | _ | Punct | Punct | _ | 2 | punct | 2 | punct | | + | |
- | + | ||
- | ==== Parsing ==== | + | |
- | + | ||
- | Nonprojectivities in BTB are rare. Only 747 of the 196,151 tokens in the CoNLL 2006 version are attached nonprojectively (0.38%). | + | |
- | + | ||
- | The results of the CoNLL 2006 shared task are [[http:// | + | |
- | + | ||
- | ^ Parser (Authors) ^ LAS ^ UAS ^ | + | |
- | | MST (McDonald et al.) | 87.57 | 92.04 | | + | |
- | | Malt (Nivre et al.) | 87.41 | 91.72 | | + | |
- | | Nara (Yuchang Cheng) | 86.34 | 91.30 | | + | |
- | + | ||
- | ===== Bengali (bn) ===== | + | |
- | + | ||
- | Hyderabad Dependency Treebank (HyDT-Bangla) | + | |
- | + | ||
- | ==== Versions ==== | + | |
- | + | ||
- | | + | |
- | | + | |
- | * CoNLL format | + | |
- | * ICON 2010 | + | |
- | * Shakti Standard Format (SSF; native) | + | |
- | * CoNLL format | + | |
- | + | ||
- | There has been no official release of the treebank yet. There have been two as-is sample releases for the purposes of the NLP tools contests in parsing Indian languages, attached to the [[http:// | + | |
- | + | ||
- | ==== Obtaining and License ==== | + | |
- | + | ||
- | There is no standard distribution channel for the treebank after the ICON 2010 evaluation period. Inquire at the LTRC (ltrc (at) iiit (dot) ac (dot) in) about the possibility of getting the data. The ICON 2010 license in short: | + | |
- | + | ||
- | * non-commercial research usage | + | |
- | * no redistribution | + | |
- | + | ||
- | HyDT-Bangla is being created by members of the [[http:// | + | |
- | + | ||
- | ==== Domain ==== | + | |
- | + | ||
- | Unknown. | + | |
- | + | ||
- | ==== Size ==== | + | |
- | + | ||
- | HyDT-Bangla shows dependencies between chunks, not words. The node/tree ratio is thus much lower than in other treebanks. The ICON 2009 version came with a data split into three parts: training, development and test: | + | |
- | + | ||
- | ^ Part ^ Sentences ^ Chunks ^ Ratio ^ | + | |
- | | Training | 980 | 6449 | 6.58 | | + | |
- | | Development | 150 | 811 | 5.41 | | + | |
- | | Test | 150 | 961 | 6.41 | | + | |
- | | TOTAL | 1280 | 8221 | 6.42 | | + | |
- | + | ||
- | The ICON 2010 version came with a data split into three parts: training, development and test: | + | |
- | + | ||
- | ^ Part ^ Sentences ^ Chunks ^ Ratio ^ Words ^ Ratio ^ | + | |
- | | Training | 979 | 6440 | 6.58 | 10305 | 10.52 | | + | |
- | | Development | 150 | 812 | 5.41 | 1196 | 7.97 | | + | |
- | | Test | 150 | 961 | 6.41 | 1350 | 9.00 | | + | |
- | | TOTAL | 1279 | 8213 | 6.42 | 12851 | 10.04 | | + | |
- | + | ||
- | I have counted the sentences and chunks. The number of words comes from (Husain et al., 2010). Note that the paper gives the number of training sentences as 980 (instead of 979), which is a mistake. The last training sentence has the id 980 but there is no sentence with id 418. | + | |
- | + | ||
- | Apparently the training-development-test data split was more or less identical in both years, except for the minor discrepancies (number of training sentences and development chunks). | + | |
- | + | ||
- | ==== References ==== | + | |
- | + | ||
- | * Website | + | |
- | * //no website dedicated to the treebank// | + | |
- | * [[http:// | + | |
- | * [[http:// | + | |
- | * Data | + | |
- | * //no separate citation// | + | |
- | * Principal publications | + | |
- | * Samar Husain: | + | |
- | * Samar Husain, Prashanth Mannem, Bharat Ambati, Phani Gadde: //The ICON-2010 tools contest on Indian language dependency parsing.// In: [[http:// | + | |
- | * Documentation | + | |
- | * http:// | + | |
- | + | ||
- | ==== Inside ==== | + | |
- | + | ||
- | The text uses the [[http:// | + | |
- | + | ||
- | The CoNLL format contains only the chunk heads. The native SSF format shows the other words in the chunk, too, but it does not capture intra-chunk dependency relations. This is an example of a multi-word chunk: | + | |
- | + | ||
- | < | + | |
- | 3.1 | + | |
- | 3.2 | + | |
- | 3.3 | + | |
- | ))</ | + | |
- | + | ||
- | In the CoNLL format, the CPOS column contains the [[http:// | + | |
- | + | ||
- | Occasionally there are '' | + | |
- | + | ||
- | The [[http:// | + | |
- | + | ||
- | According to [[http:// | + | |
- | + | ||
- | Note: There have been cycles in the Hindi part of HyDT but no such problem occurs in the Bengali part. | + | |
- | + | ||
- | ==== Sample ==== | + | |
- | + | ||
- | The first sentence of the ICON 2010 training data (with fine-grained syntactic tags) in the Shakti format: | + | |
- | + | ||
- | <code xml>< | + | |
- | < | + | |
- | < | + | |
- | < | + | |
- | < | + | |
- | < | + | |
- | < | + | |
- | < | + | |
- | </ | + | |
- | </ | + | |
- | </ | + | |
- | < | + | |
- | 1 (( NP < | + | |
- | 1.1 mudZira NN < | + | |
- | 1.2 Agei NST < | + | |
- | )) | + | |
- | 2 (( NP < | + | |
- | 2.1 praWama QO < | + | |
- | 2.2 kApa NN < | + | |
- | 2.3 cA NN < | + | |
- | )) | + | |
- | 3 (( VGF < | + | |
- | 3.1 ese VM < | + | |
- | 3.2 . SYM < | + | |
- | )) | + | |
- | </ | + | |
- | + | ||
- | And in the CoNLL format: | + | |
- | + | ||
- | | 1 | Agei | Age | NP | NST | lex-Age< | + | |
- | | 2 | cA | cA | NP | NN | lex-cA< | + | |
- | | 3 | ese | As | VGF | VM | lex-As< | + | |
- | + | ||
- | And after conversion of the WX encoding to the Bengali script in UTF-8: | + | |
- | + | ||
- | | 1 | আগেই | আগে | NP | NST | lex-Age< | + | |
- | | 2 | চা | চা | NP | NN | lex-cA< | + | |
- | | 3 | এসে | আস্ | VGF | VM | lex-As< | + | |
- | + | ||
- | The first sentence of the ICON 2010 development data (with fine-grained syntactic tags) in the Shakti format: | + | |
- | + | ||
- | <code xml>< | + | |
- | < | + | |
- | < | + | |
- | < | + | |
- | < | + | |
- | < | + | |
- | < | + | |
- | < | + | |
- | </ | + | |
- | </ | + | |
- | </ | + | |
- | < | + | |
- | 1 (( NP < | + | |
- | 1.1 parabarwIkAle NN < | + | |
- | )) | + | |
- | 2 (( NP < | + | |
- | 2.1 aPisa-biyArAraxera NN < | + | |
- | )) | + | |
- | 3 (( NP < | + | |
- | 3.1 nAma NN < | + | |
- | )) | + | |
- | 4 (( NP < | + | |
- | 4.1 GoRaNA NN < | + | |
- | )) | + | |
- | 5 (( VGNN < | + | |
- | 5.1 karAra VM < | + | |
- | )) | + | |
- | 6 (( NP < | + | |
- | 6.1 samay NN < | + | |
- | )) | + | |
- | 7 (( NP < | + | |
- | 7.1 animeRake NNP < | + | |
- | )) | + | |
- | 8 (( VGF < | + | |
- | 8.1 sariye VM < | + | |
- | 8.2 . SYM < | + | |
- | )) | + | |
- | </ | + | |
- | + | ||
- | And in the CoNLL format: | + | |
- | + | ||
- | | 1 | parabarwIkAle | parabarwIkAle | NP | NN | lex-parabarwIkAle< | + | |
- | | 2 | aPisa-biyArAraxera | aPisa-biyArAraxera | NP | NN | lex-aPisa-biyArAraxera< | + | |
- | | 3 | nAma | nAma | NP | NN | lex-nAma< | + | |
- | | 4 | GoRaNA | GoRaNA | NP | NN | lex-GoRaNA< | + | |
- | | 5 | karAra | kar | VGNN | VM | lex-kar< | + | |
- | | 6 | samay | samay | NP | NN | lex-samay< | + | |
- | | 7 | animeRake | animeRake | NP | NNP | lex-animeRake< | + | |
- | | 8 | sariye | sariye | VGF | VM | lex-sariye< | + | |
- | + | ||
- | And after conversion of the WX encoding to the Bengali script in UTF-8: | + | |
- | + | ||
- | | 1 | পরবর্তীকালে | পরবর্তীকালে | NP | NN | lex-parabarwIkAle< | + | |
- | | 2 | অফিস-বিযারারদের | অফিস-বিযারারদের | NP | NN | lex-aPisa-biyArAraxera< | + | |
- | | 3 | নাম | নাম | NP | NN | lex-nAma< | + | |
- | | 4 | ঘোষণা | ঘোষণা | NP | NN | lex-GoRaNA< | + | |
- | | 5 | করার | কর্ | VGNN | VM | lex-kar< | + | |
- | | 6 | সময্ | সময্ | NP | NN | lex-samay< | + | |
- | | 7 | অনিমেষকে | অনিমেষকে | NP | NNP | lex-animeRake< | + | |
- | | 8 | সরিযে | সরিযে | VGF | VM | lex-sariye< | + | |
- | + | ||
- | The first sentence of the ICON 2010 test data (with fine-grained syntactic tags) in the Shakti format: | + | |
- | + | ||
- | <code xml>< | + | |
- | < | + | |
- | < | + | |
- | < | + | |
- | < | + | |
- | < | + | |
- | < | + | |
- | < | + | |
- | </ | + | |
- | < | + | |
- | </ | + | |
- | < | + | |
- | 1 (( NP < | + | |
- | 1.1 mAXabIlawA NNP < | + | |
- | )) | + | |
- | 2 (( NP < | + | |
- | 2.1 waKana PRP < | + | |
- | )) | + | |
- | 3 (( NP < | + | |
- | 3.1 hAwera NN < | + | |
- | )) | + | |
- | 4 (( NP < | + | |
- | 4.1 GadZi NN < | + | |
- | )) | + | |
- | 5 (( VGNF < | + | |
- | 5.1 Kule VM < | + | |
- | )) | + | |
- | 6 (( NP < | + | |
- | 6.1 tebile NN < | + | |
- | )) | + | |
- | 7 (( VGF < | + | |
- | 7.1 rAKaCila VM < | + | |
- | 7.2 । SYM | + | |
- | )) | + | |
- | </ | + | |
- | + | ||
- | And in the CoNLL format: | + | |
- | + | ||
- | | 1 | mAXabIlawA | mAXabIlawA | NP | NNP | lex-mAXabIlawA< | + | |
- | | 2 | waKana | waKana | NP | PRP | lex-waKana< | + | |
- | | 3 | hAwera | hAwa | NP | NN | lex-hAwa< | + | |
- | | 4 | GadZi | GadZi | NP | NN | lex-GadZi< | + | |
- | | 5 | Kule | Kul | VGNF | VM | lex-Kul< | + | |
- | | 6 | tebile | tebila | NP | NN | lex-tebila< | + | |
- | | 7 | rAKaCila | rAK | VGF | VM | lex-rAK< | + | |
- | + | ||
- | And after conversion of the WX encoding to the Bengali script in UTF-8: | + | |
- | + | ||
- | | 1 | মাধবীলতা | মাধবীলতা | NP | NNP | lex-mAXabIlawA< | + | |
- | | 2 | তখন | তখন | NP | PRP | lex-waKana< | + | |
- | | 3 | হাতের | হাত | NP | NN | lex-hAwa< | + | |
- | | 4 | ঘড়ি | ঘড়ি | NP | NN | lex-GadZi< | + | |
- | | 5 | খুলে | খুল্ | VGNF | VM | lex-Kul< | + | |
- | | 6 | টেবিলে | টেবিল | NP | NN | lex-tebila< | + | |
- | | 7 | রাখছিল | রাখ্ | VGF | VM | lex-rAK< | + | |
- | + | ||
- | ==== Parsing ==== | + | |
- | + | ||
- | Nonprojectivities in HyDT-Bangla are not frequent. Only 78 of the 7252 chunks in the training+development ICON 2010 version are attached nonprojectively (1.08%). | + | |
- | + | ||
- | The results of the ICON 2009 NLP tools contest have been published in [[http:// | + | |
- | + | ||
- | ^ Parser (Authors) ^ LAS ^ UAS ^ | + | |
- | | Kolkata (De et al.)* | 84.29 | 90.32 | | + | |
- | | Hyderabad (Ambati et al.) | 78.25 | 90.22 | | + | |
- | | Malt (Nivre) | 76.07 | 88.97 | | + | |
- | | Malt+MST (Zeman) | 71.49 | 86.89 | | + | |
- | | Mannem | 70.34 | 83.56 | | + | |
- | + | ||
- | The results of the ICON 2010 NLP tools contest have been published in [[http:// | + | |
- | + | ||
- | ^ Parser (Authors) ^ LAS ^ UAS ^ | + | |
- | | Attardi et al. | 70.66 | 87.41 | | + | |
- | | Kosaraju et al. | 70.55 | 86.16 | | + | |
- | | Kolachina et al. | 70.14 | 87.10 | | + | |
- | + | ||
- | ===== Catalan | + | |
- | + | ||
- | There is [[http:// | + | |
- | * CESS-Cat | + | |
- | * Cat3LB | + | |
- | * AnCora-CA | + | |
- | + | ||
- | ==== Versions ==== | + | |
- | + | ||
- | * CoNLL 2007 (CESS-Cat) | + | |
- | * CoNLL 2009 (AnCora-CA) | + | |
- | + | ||
- | The dependency treebank Cat3LB was extracted automatically from an earlier constituent-based annotation (see Montserrat Civit, Ma. Antònia Martí, Núria Bufí: | + | |
- | + | ||
- | ==== Obtaining and License ==== | + | |
- | + | ||
- | The AnCora-CA corpus ought to be freely downloadable from [[http:// | + | |
- | + | ||
- | Republication of the two CoNLL versions in LDC is planned but it has not happenned yet. | + | |
- | + | ||
- | The CoNLL 2007 license in short: | + | |
- | + | ||
- | * research and demonstrative usage | + | |
- | * no redistribution | + | |
- | * cite in publications | + | |
- | * The original CoNLL 2007 license required a reference to the CESS-ECE // | + | |
- | * Later there was [[http:// | + | |
- | + | ||
- | AnCora-CA was created by members of the [[http:// | + | |
- | + | ||
- | ==== Domain ==== | + | |
- | + | ||
- | Mostly newswire (EFE news, ACN Catalan news, Catalan version of El Periódico, 2000). | + | |
- | + | ||
- | ==== Size ==== | + | |
- | + | ||
- | The CoNLL 2007 version contains 435,860 tokens in 15125 sentences, yielding 28.82 tokens per sentence on average (CoNLL 2007 data split: 430,844 tokens / 14958 sentences training, 5016 tokens / 167 sentences test). | + | |
- | + | ||
- | The CoNLL 2009 version contains 496,672 tokens in 16786 sentences, yielding 29.59 tokens per sentence on average (CoNLL 2009 data split: 390,302 tokens / 13200 sentences training, 53015 tokens / 1724 sentences development, | + | |
- | + | ||
- | ==== References ==== | + | |
- | + | ||
- | * Website | + | |
- | * http:// | + | |
- | * Data | + | |
- | * //no separate citation// | + | |
- | * Principal publications | + | |
- | * Mariona Taulé, M. Antònia Martí, Marta Recasens: | + | |
- | * Documentation | + | |
- | * Maria Anntònia Martí, Mariona Taulé, Manu Bertran, Lluís Màrquez: | + | |
- | * [[http:// | + | |
- | * [[http:// | + | |
- | + | ||
- | ==== Inside ==== | + | |
- | + | ||
- | The original morphosyntactic tags (EAGLES?) have been converted to fit into the three columns (CPOS, POS and FEAT) columns of the CoNLL 2006/7 format, resp. the two columns (POS and FEAT) of the CoNLL 2009 format. Note that the missing CPOS column is not the only difference between the two conversion schemes. [[http:// | + | |
- | + | ||
- | The morphosyntactic tags have been disambiguated manually. The CoNLL 2009 version also contains automatically disambiguated tags. | + | |
- | + | ||
- | Multi-word expressions have been collapsed into one token, using underscore as the joining character. This includes named entities (e.g. La_Garrotxa, | + | |
- | + | ||
- | ==== Sample ==== | + | |
- | + | ||
- | The first sentence of the CoNLL 2007 training data: | + | |
- | + | ||
- | | 1 | L' | el | d | da | num=s< | + | |
- | | 2 | Ajuntament_de_Manresa | Ajuntament_de_Manresa | n | np | _ | 4 | SUJ | _ | _ | | + | |
- | | 3 | ha | haver | v | va | num=s< | + | |
- | | 4 | posat_en_funcionament | posar_en_funcionament | v | vm | num=s< | + | |
- | | 5 | tot | tot | d | di | num=s< | + | |
- | | 6 | un_seguit_de | un_seguit_de | d | di | num=p< | + | |
- | | 7 | mesures | mesura | n | nc | num=p< | + | |
- | | 8 | , | , | F | Fc | _ | 10 | PUNC | _ | _ | | + | |
- | | 9 | la | el | d | da | num=s< | + | |
- | | 10 | majoria | majoria | n | nc | num=s< | + | |
- | | 11 | informatives | informatiu | a | aq | num=p< | + | |
- | | 12 | , | , | F | Fc | _ | 10 | PUNC | _ | _ | | + | |
- | | 13 | que | que | p | pr | num=n< | + | |
- | | 14 | tenen | tenir | v | vm | num=p< | + | |
- | | 15 | com_a | com_a | s | sp | for=s | 14 | CPRED | _ | _ | | + | |
- | | 16 | finalitat | finalitat | n | nc | num=s< | + | |
- | | 17 | minimitzar | minimitzar | v | vm | mod=n | 14 | CD | _ | _ | | + | |
- | | 18 | els | el | d | da | num=p< | + | |
- | | 19 | efectes | efecte | n | nc | num=p< | + | |
- | | 20 | de | de | s | sp | for=s | 19 | SP | _ | _ | | + | |
- | | 21 | la | el | d | da | num=s< | + | |
- | | 22 | vaga | vaga | n | nc | num=s< | + | |
- | | 23 | . | . | F | Fp | _ | 4 | PUNC | _ | _ | | + | |
- | + | ||
- | The first sentence of the CoNLL 2007 test data: | + | |
- | + | ||
- | | 1 | Tot_i_que | tot_i_que | c | cs | _ | 5 | SUBORD | _ | _ | | + | |
- | | 2 | ahir | ahir | r | rg | _ | 5 | CC | _ | _ | | + | |
- | | 3 | hi | hi | p | pp | num=n< | + | |
- | | 4 | va | anar | v | va | num=s< | + | |
- | | 5 | haver | haver | v | va | mod=n | 15 | AO | _ | _ | | + | |
- | | 6 | una | un | d | di | num=s< | + | |
- | | 7 | reunió | reunió | n | nc | num=s< | + | |
- | | 8 | de | de | s | sp | for=s | 7 | SP | _ | _ | | + | |
- | | 9 | darrera | darrer | a | ao | num=s< | + | |
- | | 10 | hora | hora | n | nc | num=s< | + | |
- | | 11 | , | , | F | Fc | _ | 5 | PUNC | _ | _ | | + | |
- | | 12 | no | no | r | rn | _ | 15 | MOD | _ | _ | | + | |
- | | 13 | es | es | p | p0 | _ | 15 | PASS | _ | _ | | + | |
- | | 14 | va | anar | v | va | num=s< | + | |
- | | 15 | aconseguir | aconseguir | v | vm | mod=n | 0 | S | _ | _ | | + | |
- | | 16 | acostar | acostar | v | vm | mod=n | 15 | SUJ | _ | _ | | + | |
- | | 17 | posicions | posició | n | nc | num=p< | + | |
- | | 18 | , | , | F | Fc | _ | 23 | PUNC | _ | _ | | + | |
- | | 19 | de_manera_que | de_manera_que | c | cs | _ | 23 | SUBORD | _ | _ | | + | |
- | | 20 | els | el | d | da | num=p< | + | |
- | | 21 | treballadors | treballador | n | nc | num=p< | + | |
- | | 22 | han | haver | v | va | num=p< | + | |
- | | 23 | decidit | decidir | v | vm | num=s< | + | |
- | | 24 | anar | anar | v | vm | mod=n | 23 | CD | _ | _ | | + | |
- | | 25 | a | a | s | sp | for=s | 24 | CREG | _ | _ | | + | |
- | | 26 | la | el | d | da | num=s< | + | |
- | | 27 | vaga | vaga | n | nc | num=s< | + | |
- | | 28 | . | . | F | Fp | _ | 15 | PUNC | _ | _ | | + | |
- | + | ||
- | ==== Parsing ==== | + | |
- | + | ||
- | Nonprojectivities in BTB are rare. Only 747 of the 196,151 tokens in the CoNLL 2006 version are attached nonprojectively | + | |
- | + | ||
- | The results of the CoNLL 2006 shared task are [[http:// | + | |
- | + | ||
- | ^ Parser (Authors) ^ LAS ^ UAS ^ | + | |
- | | MST (McDonald et al.) | 87.57 | 92.04 | | + | |
- | | Malt (Nivre et al.) | 87.41 | 91.72 | | + | |
- | | Nara (Yuchang Cheng) | 86.34 | 91.30 | | + |