[ Skip to the content ]

Institute of Formal and Applied Linguistics Wiki


[ Back to the navigation ]

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision Both sides next revision
external:spr [2011/10/21 14:11]
smejkalova
external:spr [2011/10/21 14:32]
smejkalova
Line 31: Line 31:
  
  
-Here we present a small portion of the data just to show its structure.+Here we present a small portion of the data just to show its structure. ​For each verb we define a set of semantic patterns and provide a  sample of manually annotated corpus concordances. Then we do a detailed interannotator disagreement analysis, error detection and adjudication. 
  
  
 ==== Annotation Scheme Description ==== ==== Annotation Scheme Description ====
-     ​* ​Specification of form for defining patterns ​- [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​PDEV2.1-pattern-form.pdf|pdf]]+     ​* ​Pattern Definition Form - [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​PDEV2.1-pattern-form.pdf|pdf]]
      * Annotation Manual - [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​annotation_manual.pdf|pdf]]      * Annotation Manual - [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​annotation_manual.pdf|pdf]]
 +
 +
 +==== Pattern Definitions Preview ====
 +
 +Pattern definitions - we have revised the pattern definitions for 30 verbs. Here is the sample of three of them (after revision):
 +     * [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​cool_patterns.html|cool]]
 +     * [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​deny_patterns.html|deny]] - detailed [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​deny_7.png|view]] of pattern number 7
 +     * [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​yield_patterns.html|yield]]
 +
  
  
 ==== Annotated Data Preview ==== ==== Annotated Data Preview ====
 +Annotation of 50 concordances per each of these verbs by three annotators. Here is a little sample of three verbs. There has been already finished the manual disagreements analysis and all instances were adjudicated and gold sample was created. Manual disagreements analysis and adjudication:​
 +    * cool - [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​cool_analysis.pdf|pdf]]
 +    * deny - [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​deny_analysis.pdf|pdf]]
 +    * yield - [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​yield_analysis.pdf|pdf]]
  
  
 ==== Disagreement Analysis Preview ==== ==== Disagreement Analysis Preview ====
 +
 +Automatic disagreements analysis - confusion matrix for each pair of the annotators in one file
 +    * cool - [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​cool_res.txt|txt]]
 +    * deny - [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​deny_res.txt|txt]]
 +    * yield - [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​yield_res.txt|txt]]
  
  
-     * Pattern definitions - we have revised the pattern definitions for 30 verbs. Here is the sample of three of them (after revision): 
-          * [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​cool_patterns.html|cool]] 
-          * [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​deny_patterns.html|deny]] - detailed [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​deny_7.png|view]] of pattern number 7 
-          * [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​yield_patterns.html|yield]] 
-     * Annotation of 50 concordances per each of these verbs by three annotators. Here is a little sample of three verbs. There has been already finished the manual disagreements analysis and all instances were adjudicated and gold sample was created. 
-          * Manual disagreements analysis and adjudication:​ 
-                * cool - [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​cool_analysis.pdf|pdf]] 
-                * deny - [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​deny_analysis.pdf|pdf]] 
-                * yield - [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​yield_analysis.pdf|pdf]] 
-          * Automatic disagreements analysis - confusion matrix for each pair of the annotators in one file 
-                * cool - [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​cool_res.txt|txt]] 
-                * deny - [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​deny_res.txt|txt]] 
-                * yield - [[http://​ufal.mff.cuni.cz/​~smejkalova/​pdev/​yield_res.txt|txt]] 
-          * Inter-annotator agreement (Cohen'​s kappa for each pair, Fleiss'​ kappa for all together 
  
-Results of inter-annotator agreement 
-^verb     ​^ ​ size  ^ #N ^ Fleiss'​ kappa ^        Cohen'​s kappa            ^^^ 
-|         ​| ​       |    |               | A2 vs. A3 | A2 vs. A1 | A3 vs. A1 | 
-|cool   |      50|  16|    ​0.685| ​     0.743| ​     0.669| ​     0.646| 
-|deny   |      50|  10|          0.524| ​     0.434| ​     0.571| ​     0.582| 
-|yield ​   |      50|  10|    ​0.500| ​     0.489| ​     0.588| ​     0.429| 
  
  

[ Back to the navigation ] [ Back to the content ]