[ Skip to the content ]

Institute of Formal and Applied Linguistics Wiki

[ Back to the navigation ]

You are here: start » user » zeman » treebanks » te

This is an old revision of the document!

Table of Contents

Telugu (te)

Hyderabad Dependency Treebank (HyDT-Telugu)

Versions

There has been no official release of the treebank yet. There have been two as-is sample releases for the purposes of the NLP tools contests in parsing Indian languages, attached to the ICON 2009 and 2010 conferences.

Obtaining and License

There is no standard distribution channel for the treebank after the ICON 2010 evaluation period. Inquire at the LTRC (ltrc (at) iiit (dot) ac (dot) in) about the possibility of getting the data. The ICON 2010 license in short:

HyDT-Telugu is being created by members of the Language Technologies Research Centre, International Institute of Information Technology, Gachibowli, Hyderabad, 500032, India.

References

Domain

Unknown.

Size

HyDT-Telugu shows dependencies between chunks, not words. The node/tree ratio is thus much lower than in other treebanks. The ICON 2009 version came with a data split into three parts: training, development and test:

Part Sentences Chunks Ratio
Training 1456 5494 3.77
Development 150 675 4.50
Test 150 583 3.89
TOTAL 1756 6752 3.85

The data distributed for ICON 2010 was slightly smaller, maybe it had been cleaned up? Note that the number of training words, 7602, is identical to the number published for ICON 2009. I cannot verify it because I only see chunks, not words in the CoNLL data format.

Part Sentences Chunks Ratio Words Ratio
Training 1400 ? ? 7602 5.43
Development 150 ? ? 839 5.59
Test 150 ? ? 836 5.57
TOTAL 1700 ? ? 9277 5.46

We drew our training and test data from the ICON 2010 datasets but we have fewer sentences – why?

Part Sentences Chunks Ratio
Training 1300 5125 3.94
Test 150 597 3.98
TOTAL 1450 5722 3.95

Inside

The text uses the WX encoding of Indian letters. If we know what the original script is (Bengali in this case) we can map the WX encoding to the original characters in UTF-8. WX uses English letters so if there was embedded English (or other string using Latin letters) it will probably get lost during the conversion.

The CoNLL format contains only the chunk heads. The native SSF format shows the other words in the chunk, too, but it does not capture intra-chunk dependency relations. This is an example of a multi-word chunk: 

3       ((      NP      <fs af='rumAla,n,,sg,,d,0,0' head="rumAla" drel=k2:VGF name=NP3>
3.1     ekatA   QC      <fs af='eka,num,,,,,,'>
3.2     ledisa  JJ      <fs af='ledisa,unk,,,,,,'>
3.3     rumAla  NN      <fs af='rumAla,n,,sg,,d,0,0' name="rumAla">
        ))

In the CoNLL format, the CPOS column contains the chunk label (e.g. NP = noun phrase) and the POS column contains the part of speech of the chunk head.

Occasionally there are NULL nodes that do not correspond to any surface chunk or token. They represent ellided participants.

The syntactic tags (dependency relation labels) are karaka relations, i.e. deep syntactic roles according to the Pāṇinian grammar. There are separate versions of the treebank with fine-grained and coarse-grained syntactic tags.

According to (Husain et al., 2010), in the ICON 2010 version, the chunk tags, POS tags and inter-chunk dependencies (topology + tags) were annotated manually. The rest (lemma, morphosyntactic features, headword of chunk) was marked automatically.

Note: There have been cycles in the Hindi part of HyDT but no such problem occurs in the Bengali part.

Sample

The first sentence of the ICON 2010 training data (with fine-grained syntactic tags) in the Shakti format: 

<document id="">			
<head>
<annotated-resource name="HyDT-Bangla" version="0.5" type="dep-interchunk-only" layers="morph,pos,chunk,dep-interchunk-only" language="ben" date-of-release="20100831">
    <annotation-standard>
        <morph-standard name="Anncorra-morph" version="1.31" date="20080920" />
        <pos-standard name="Anncorra-pos" version="" date="20061215" />
        <chunk-standard name="Anncorra-chunk" version="" date="20061215" />
        <dependency-standard name="Anncorra-dep" version="2.0" date="" dep-tagset-granularity="6" />
    </annotation-standard>
</annotated-resource>		
</head>			
<Sentence id="1">
1	((	NP	<fs af='Age,adv,,,,,,' head="Agei" drel=k7t:VGF name=NP>
1.1	mudZira	NN	<fs af='mudZi,n,,sg,,o,era,era'>
1.2	Agei	NST	<fs af='Age,adv,,,,,,' name="Agei">
	))		
2	((	NP	<fs af='cA,n,,sg,,d,0,0' head="cA" drel=k1:VGF name=NP2>
2.1	praWama	QO	<fs af='praWama,num,,,,,,'>
2.2	kApa	NN	<fs af='kApa,unk,,,,,,'>
2.3	cA	NN	<fs af='cA,n,,sg,,d,0,0' name="cA">
	))		
3	((	VGF	<fs af='As,v,,,5,,A_yA+Ce,A' head="ese" name=VGF>
3.1	ese	VM	<fs af='As,v,,,7,,A,A' name="ese">
3.2	.	SYM	<fs af='.,punc,,,,,,'>
	))		
</Sentence>

And in the CoNLL format:

1 Agei Age NP NST lex-Age|cat-adv|gend-|num-|pers-|case-|vib-|tam-|head-Agei|name-NP 3 k7t _ _
2 cA cA NP NN lex-cA|cat-n|gend-|num-sg|pers-|case-d|vib-0|tam-0|head-cA|name-NP2 3 k1 _ _
3 ese As VGF VM lex-As|cat-v|gend-|num-|pers-5|case-|vib-A_yA+Ce|tam-A|head-ese|name-VGF 0 main _ _

And after conversion of the WX encoding to the Bengali script in UTF-8:

1 আগেই আগে NP NST lex-Age|cat-adv|gend-|num-|pers-|case-|vib-|tam-|head-Agei|name-NP 3 k7t _ _
2 চা চা NP NN lex-cA|cat-n|gend-|num-sg|pers-|case-d|vib-0|tam-0|head-cA|name-NP2 3 k1 _ _
3 এসে আস্ VGF VM lex-As|cat-v|gend-|num-|pers-5|case-|vib-A_yA+Ce|tam-A|head-ese|name-VGF 0 main _ _

The first sentence of the ICON 2010 development data (with fine-grained syntactic tags) in the Shakti format: 

<document id="">
<head>
<annotated-resource name="HyDT-Bangla" version="0.5" type="dep-interchunk-only" layers="morph,pos,chunk,dep-interchunk-only" language="ben" date-of-release="20100831">
    <annotation-standard>
        <morph-standard name="Anncorra-morph" version="1.31" date="20080920" />
        <pos-standard name="Anncorra-pos" version="" date="20061215" />
        <chunk-standard name="Anncorra-chunk" version="" date="20061215" />
        <dependency-standard name="Anncorra-dep" version="2.0" date="" dep-tagset-granularity="6" />
    </annotation-standard>
</annotated-resource>
</head>
<Sentence id="1">
1	((	NP	<fs af='parabarwIkAle,adv,,,,,,' head="parabarwIkAle" drel=k7t:VGF name=NP>
1.1	parabarwIkAle	NN	<fs af='parabarwIkAle,adv,,,,,,' name="parabarwIkAle">
	))		
2	((	NP	<fs af='aPisa-biyArAraxera,unk,,,,,,' head="aPisa-biyArAraxera" drel=r6:NP3 name=NP2>
2.1	aPisa-biyArAraxera	NN	<fs af='aPisa-biyArAraxera,unk,,,,,,' name="aPisa-biyArAraxera">
	))		
3	((	NP	<fs af='nAma,n,,sg,,d,0,0' head="nAma" drel=k2:VGNN name=NP3>
3.1	nAma	NN	<fs af='nAma,n,,sg,,d,0,0' name="nAma">
	))		
4	((	NP	<fs af='GoRaNA,unk,,,,,,' head="GoRaNA" drel=pof:VGNN name=NP4>
4.1	GoRaNA	NN	<fs af='GoRaNA,unk,,,,,,' name="GoRaNA">
	))		
5	((	VGNN	<fs af='kar,n,,,any,,,' head="karAra" drel=r6:NP5 name=VGNN>
5.1	karAra	VM	<fs af='kar,n,,,any,,,' name="karAra">
	))		
6	((	NP	<fs af='samay,unk,,,,,,' head="samay" drel=k7t:VGF name=NP5>
6.1	samay	NN	<fs af='samay,unk,,,,,,' name="samay">
	))		
7	((	NP	<fs af='animeRake,unk,,,,,,' head="animeRake" drel=k2:VGF name=NP6>
7.1	animeRake	NNP	<fs af='animeRake,unk,,,,,,' name="animeRake">
	))		
8	((	VGF	<fs af='sariye,unk,,,5,,0_rAKA+ka_ha+la,' head="sariye" name=VGF>
8.1	sariye	VM	<fs af='sariye,unk,,,,,,' name="sariye">
8.2	.	SYM	<fs af='.,punc,,,,,,'>
	))		
</Sentence>

And in the CoNLL format:

1 parabarwIkAle parabarwIkAle NP NN lex-parabarwIkAle|cat-adv|gend-|num-|pers-|case-|vib-|tam-|head-parabarwIkAle|name-NP 8 k7t _ _
2 aPisa-biyArAraxera aPisa-biyArAraxera NP NN lex-aPisa-biyArAraxera|cat-unk|gend-|num-|pers-|case-|vib-|tam-|head-aPisa-biyArAraxera|name-NP2 3 r6 _ _
3 nAma nAma NP NN lex-nAma|cat-n|gend-|num-sg|pers-|case-d|vib-0|tam-0|head-nAma|name-NP3 5 k2 _ _
4 GoRaNA GoRaNA NP NN lex-GoRaNA|cat-unk|gend-|num-|pers-|case-|vib-|tam-|head-GoRaNA|name-NP4 5 pof _ _
5 karAra kar VGNN VM lex-kar|cat-n|gend-|num-|pers-any|case-|vib-|tam-|head-karAra|name-VGNN 6 r6 _ _
6 samay samay NP NN lex-samay|cat-unk|gend-|num-|pers-|case-|vib-|tam-|head-samay|name-NP5 8 k7t _ _
7 animeRake animeRake NP NNP lex-animeRake|cat-unk|gend-|num-|pers-|case-|vib-|tam-|head-animeRake|name-NP6 8 k2 _ _
8 sariye sariye VGF VM lex-sariye|cat-unk|gend-|num-|pers-5|case-|vib-0_rAKA+ka_ha+la|tam-|head-sariye|name-VGF 0 main _ _

And after conversion of the WX encoding to the Bengali script in UTF-8:

1 পরবর্তীকালে পরবর্তীকালে NP NN lex-parabarwIkAle|cat-adv|gend-|num-|pers-|case-|vib-|tam-|head-parabarwIkAle|name-NP 8 k7t _ _
2 অফিস-বিযারারদের অফিস-বিযারারদের NP NN lex-aPisa-biyArAraxera|cat-unk|gend-|num-|pers-|case-|vib-|tam-|head-aPisa-biyArAraxera|name-NP2 3 r6 _ _
3 নাম নাম NP NN lex-nAma|cat-n|gend-|num-sg|pers-|case-d|vib-0|tam-0|head-nAma|name-NP3 5 k2 _ _
4 ঘোষণা ঘোষণা NP NN lex-GoRaNA|cat-unk|gend-|num-|pers-|case-|vib-|tam-|head-GoRaNA|name-NP4 5 pof _ _
5 করার কর্ VGNN VM lex-kar|cat-n|gend-|num-|pers-any|case-|vib-|tam-|head-karAra|name-VGNN 6 r6 _ _
6 সময্ সময্ NP NN lex-samay|cat-unk|gend-|num-|pers-|case-|vib-|tam-|head-samay|name-NP5 8 k7t _ _
7 অনিমেষকে অনিমেষকে NP NNP lex-animeRake|cat-unk|gend-|num-|pers-|case-|vib-|tam-|head-animeRake|name-NP6 8 k2 _ _
8 সরিযে সরিযে VGF VM lex-sariye|cat-unk|gend-|num-|pers-5|case-|vib-0_rAKA+ka_ha+la|tam-|head-sariye|name-VGF 0 main _ _

The first sentence of the ICON 2010 test data (with fine-grained syntactic tags) in the Shakti format: 

<document id="">
<head>
<annotated-resource name="HyDT-Bangla" version="0.5" type="dep-interchunk-only" layers="morph,pos,chunk,dep-interchunk-only" language="ben" date-of-release="20101013">
    <annotation-standard>
        <morph-standard name="Anncorra-morph" version="1.31" date="20080920" />
	<pos-standard name="Anncorra-pos" version="" date="20061215" />
	<chunk-standard name="Anncorra-chunk" version="" date="20061215" />
	<dependency-standard name="Anncorra-dep" version="2.0" date="" dep-tagset-granularity="6" />
    </annotation-standard>
<annotated-resource>
</head>
<Sentence id="1">
1	((	NP	<fs af='mAXabIlawA,n,,sg,,d,0,0' head="mAXabIlawA" drel=k1:VGF name=NP>
1.1	mAXabIlawA	NNP	<fs af='mAXabIlawA,n,,sg,,d,0,0' name="mAXabIlawA">
	))		
2	((	NP	<fs af='waKana,pn,,,,d,0,0' head="waKana" drel=k7t:VGF name=NP2>
2.1	waKana	PRP	<fs af='waKana,pn,,,,d,0,0' name="waKana">
	))		
3	((	NP	<fs af='hAwa,n,,sg,,o,era,era' head="hAwera" drel=r6:NP4 name=NP3>
3.1	hAwera	NN	<fs af='hAwa,n,,sg,,o,era,era' name="hAwera">
	))		
4	((	NP	<fs af='GadZi,unk,,,,,,' head="GadZi" drel=k2:VGNF name=NP4>
4.1	GadZi	NN	<fs af='GadZi,unk,,,,,,' name="GadZi">
	))		
5	((	VGNF	<fs af='Kul,v,,,5,,ne,ne' head="Kule" drel=vmod:VGF name=VGNF>
5.1	Kule	VM	<fs af='Kul,v,,,5,,ne,ne' name="Kule">
	))		
6	((	NP	<fs af='tebila,n,,sg,,d,me,me' head="tebile" drel=k7p:VGF name=NP5>
6.1	tebile	NN	<fs af='tebila,n,,sg,,d,me,me' name="tebile">
	))		
7	((	VGF	<fs af='rAK,v,,,5,,Cila,Cila' head="rAKaCila" name=VGF>
7.1	rAKaCila	VM	<fs af='rAK,v,,,5,,Cila,Cila' name="rAKaCila">
7.2	।	SYM	
	))		
</Sentence>

And in the CoNLL format:

1 mAXabIlawA mAXabIlawA NP NNP lex-mAXabIlawA|cat-n|gend-|num-sg|pers-|case-d|vib-0|tam-0|head-mAXabIlawA|name-NP 7 k1 _ _
2 waKana waKana NP PRP lex-waKana|cat-pn|gend-|num-|pers-|case-d|vib-0|tam-0|head-waKana|name-NP2 7 k7t _ _
3 hAwera hAwa NP NN lex-hAwa|cat-n|gend-|num-sg|pers-|case-o|vib-era|tam-era|head-hAwera|name-NP3 4 r6 _ _
4 GadZi GadZi NP NN lex-GadZi|cat-unk|gend-|num-|pers-|case-|vib-|tam-|head-GadZi|name-NP4 5 k2 _ _
5 Kule Kul VGNF VM lex-Kul|cat-v|gend-|num-|pers-5|case-|vib-ne|tam-ne|head-Kule|name-VGNF 7 vmod _ _
6 tebile tebila NP NN lex-tebila|cat-n|gend-|num-sg|pers-|case-d|vib-me|tam-me|head-tebile|name-NP5 7 k7p _ _
7 rAKaCila rAK VGF VM lex-rAK|cat-v|gend-|num-|pers-5|case-|vib-Cila|tam-Cila|head-rAKaCila|name-VGF 0 main _ _

And after conversion of the WX encoding to the Bengali script in UTF-8:

1 মাধবীলতা মাধবীলতা NP NNP lex-mAXabIlawA|cat-n|gend-|num-sg|pers-|case-d|vib-0|tam-0|head-mAXabIlawA|name-NP 7 k1 _ _
2 তখন তখন NP PRP lex-waKana|cat-pn|gend-|num-|pers-|case-d|vib-0|tam-0|head-waKana|name-NP2 7 k7t _ _
3 হাতের হাত NP NN lex-hAwa|cat-n|gend-|num-sg|pers-|case-o|vib-era|tam-era|head-hAwera|name-NP3 4 r6 _ _
4 ঘড়ি ঘড়ি NP NN lex-GadZi|cat-unk|gend-|num-|pers-|case-|vib-|tam-|head-GadZi|name-NP4 5 k2 _ _
5 খুলে খুল্ VGNF VM lex-Kul|cat-v|gend-|num-|pers-5|case-|vib-ne|tam-ne|head-Kule|name-VGNF 7 vmod _ _
6 টেবিলে টেবিল NP NN lex-tebila|cat-n|gend-|num-sg|pers-|case-d|vib-me|tam-me|head-tebile|name-NP5 7 k7p _ _
7 রাখছিল রাখ্ VGF VM lex-rAK|cat-v|gend-|num-|pers-5|case-|vib-Cila|tam-Cila|head-rAKaCila|name-VGF 0 main _ _

Parsing

Nonprojectivities in HyDT-Bangla are not frequent. Only 78 of the 7252 chunks in the training+development ICON 2010 version are attached nonprojectively (1.08%).

The results of the ICON 2009 NLP tools contest have been published in (Husain, 2009). There were two evaluation rounds, the first with the coarse-grained syntactic tags, the second with the fine-grained syntactic tags. To reward language independence, only systems that parsed all three languages were officially ranked. The following table presents the Bengali/coarse-grained results of the four officially ranked systems, and the best Bengali-only* system.

Parser (Authors) LAS UAS
Kolkata (De et al.)* 84.29 90.32
Hyderabad (Ambati et al.) 78.25 90.22
Malt (Nivre) 76.07 88.97
Malt+MST (Zeman) 71.49 86.89
Mannem 70.34 83.56

The results of the ICON 2010 NLP tools contest have been published in (Husain et al., 2010), page 6. These are the best results for Bengali with fine-grained syntactic tags:

Parser (Authors) LAS UAS
Attardi et al. 70.66 87.41
Kosaraju et al. 70.55 86.16
Kolachina et al. 70.14 87.10

[ Back to the navigation ] [ Back to the content ]