[ Skip to the content ]

Institute of Formal and Applied Linguistics Wiki


[ Back to the navigation ]

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
external:spr [2011/10/21 13:41]
smejkalova
external:spr [2012/01/19 10:47] (current)
smejkalova
Line 5: Line 5:
 ===== People and contacts ===== ===== People and contacts =====
  
-    * Silvie Cinková cinkova@ufal.mff.cuni.cz +    * Silvie Cinková ​<cinkova ​(at) ufal.mff.cuni.cz> 
-    * Martin Holub holub@ufal.mff.cuni.cz +    * Martin Holub <holub (at) ufal.mff.cuni.cz> 
-    * Lenka Smejkalová smejkalova@ufal.mff.cuni.cz +    * Lenka Smejkalová ​<smejkalova ​(at) ufal.mff.cuni.cz> 
-    * Vincent Kríž vincent.kriz@gmail.com+    * Vincent Kríž ​<vincent.kriz ​(at) gmail.com>
  
 Institute of Formal and Applied Linguistics Institute of Formal and Applied Linguistics
Line 18: Line 18:
  
  
-===== Data ===== 
  
-     * Manual for Annotators - [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​annotation_manual.pdf|pdf]] 
-     * Specification of form for defining patterns - [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​PDEV2.1-pattern-form.pdf|pdf]] 
-     * Pattern definitions - we have revised the pattern definitions for 30 verbs. Here is the sample of three of them (after revision): 
-          * [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​cool_patterns.html|cool]] 
-          * [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​deny_patterns.html|deny]] - detailed [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​deny_7.png|view]] of pattern number 7 
-          * [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​yield_patterns.html|yield]] 
-     * Annotation of 50 concordances per each of these verbs by three annotators. Here is a little sample of three verbs. There has been already finished the manual disagreements analysis and all instances were adjudicated and gold sample was created. 
-          * Manual disagreements analysis and adjudication:​ 
-                * cool - [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​cool_analysis.pdf|pdf]] 
-                * deny - [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​deny_analysis.pdf|pdf]] 
-                * yield - [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​yield_analysis.pdf|pdf]] 
-          * Automatic disagreements analysis - confusion matrix for each pair of the annotators in one file 
-                * cool - [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​cool_res.txt|txt]] 
-                * deny - [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​deny_res.txt|txt]] 
-                * yield - [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​yield_res.txt|txt]] 
-          * Inter-annotator agreement (Cohen'​s kappa for each pair, Fleiss'​ kappa for all together 
  
-Results ​of inter-annotator agreement + 
-^verb     ​^ ​ size  ^ #N ^ Fleiss'​ kappa ^        Cohen'​s kappa            ^^^ +===== CPA Verb Validation Sample 30 (En) ===== 
-        ​       |    |               ​A2 vs. A3 A2 vs. A1 A3 vs. A1 + 
-|cool   |      50|  ​16|    0.685     0.743     0.669     0.646| +"CPA Verb Validation Sample 30 (En)" is a newly developed lexical resource. It contains descriptions ​of the following 30 English verbs: ​ 
-|deny   ​     50|  ​10         0.524     0.434     0.571     0.582+ 
-|yield ​        50|  10|    0.500     0.489     0.588     0.429|+|//access // |//​ally// ​  ​|//​arrive// ​ |//​breathe// ​ |//​claim// ​  | 
 +|//​cool// ​   |//​cry// ​   ​|//​crush// ​  |//​deny// ​    |//​enlarge// ​
 +|//​enlist//  ​|//​forge// ​ |//furnish// |//​hail// ​    ​|//​halt// ​   | 
 +|//​part// ​   ​|//​plug// ​  |//​plough//  ​|//​pour// ​    |//​say// ​    | 
 +|//​smash// ​  |//​smell// ​ |//​steer// ​  ​|//​submit// ​  ​|//​swell// ​  | 
 +|//​tell// ​   | //throw// |//​trouble//​ |//​wake// ​    ​|//​yield// ​  | 
 + 
 + 
 + 
 +Here we present a small portion of the data just to illustrate its structureFor each verb we define a set of semantic patterns and provide a  sample of manually annotated corpus concordances. Then we do a detailed interannotator disagreement analysis, error detection, and adjudication. 
 + 
 + 
 + 
 + 
 + 
 + 
 +==== Annotation Scheme Description ==== 
 +     * Pattern Definition Form - [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​PDEV2.1-pattern-form.pdf|pdf]] 
 +     * Annotation Manual - [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​annotation_manual.pdf|pdf]] 
 + 
 + 
 + 
 + 
 + 
 + 
 + 
 + 
 + 
 + 
 +==== Pattern Definitions Preview ==== 
 +A few examples of revised PDEV entries: 
 +     * [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​cool_patterns.html|cool]] 
 +     * [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​deny_patterns.html|deny]] 
 +     * [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​yield_patterns.html|yield]] 
 + 
 +An example of a pattern definition form in detail: 
 +     * [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​deny_7.png|deny - pattern number 7]] 
 + 
 + 
 + 
 + 
 + 
 + 
 + 
 + 
 +==== Annotated Data Preview ==== 
 +Each of the examples below contains a multiple annotated set of 50 corpus concordances with manual disagreement analysis and final adjudication. The adjudicated data are used as a gold standard sample. 
 +    * cool - [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​cool_analysis.pdf|pdf]] 
 +    * deny - [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​deny_analysis.pdf|pdf]] 
 +    * yield - [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​yield_analysis.pdf|pdf]] 
 + 
 + 
 + 
 + 
 + 
 + 
 +==== Disagreement Analysis Preview ==== 
 +Confusion matrices for each pair of annotators and automatic disagreements analysis: 
 +    * cool - [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​cool_res.txt|txt]] 
 +    * deny - [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​deny_res.txt|txt]] 
 +    * yield - [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​yield_res.txt|txt]] 
 + 
 + 
 + 
 + 
 + 
  

[ Back to the navigation ] [ Back to the content ]