• Silvie Cinková cinkova@ufal.mff.cuni.cz
• Martin Holub holub@ufal.mff.cuni.cz
• Lenka Smejkalová smejkalova@ufal.mff.cuni.cz
• Vincent Kríž vincent.kriz@gmail.com

Institute of Formal and Applied Linguistics
Charles University in Prague
Faculty of Mathematics and Physics

Malostranské náměstí 25
CZ-118 00 Praha

• Manual for Annotators - pdf
• Specification of form for defining patterns - pdf
• Pattern definitions - we have revised the pattern definitions for 30 verbs. Here is the sample of three of them (after revision):
  • cool
  • deny - detailed view of pattern number 7
  • yield
• Annotation of 50 concordances per each of these verbs by three annotators. Here is a little sample of three verbs. There has been already finished the manual disagreements analysis and all instances were adjudicated and gold sample was created.
  • Manual disagreements analysis and adjudication:
    • cool - pdf
    • deny - pdf
    • yield - pdf
  • Automatic disagreements analysis - confusion matrix for each pair of the annotators in one file
    • cool - txt
    • deny - txt
    • yield - txt
  • Inter-annotator agreement (Cohen's kappa for each pair, Fleiss' kappa for all together

Results of inter-annotator agreement