This is an old revision of the document!
Semantic Pattern Recognition
Doplnit strucny popis.
Data
* Manual for Annotators - pdf
* Specification of form for defining patterns - pdf
* Pattern definitions - we have revised the pattern definitions for 30 verbs. Here is the sample of three of them (after revision):
* Annotation of 50 concordances per each of these verbs by three annotators. Here is a little sample of three verbs. There has been already finished the manual disagreements analysis and all instances were adjudicated and gold sample was created.
- Manual disagreements analysis and adjudication:
- Automatic disagreements analysis - confusion matrix for each pair of the annotators in one file
- Inter-annotator agreement (Cohen's kappa for each pair, Fleiss' kappa for all together
Results of inter-annotator agreement
| verb | size | #N | Fleiss' kappa | Cohen's kappa | ||
|---|---|---|---|---|---|---|
| A2 vs. A3 | A2 vs. A1 | A3 vs. A1 | ||||
| cool | 50 | 16 | 0.685 | 0.743 | 0.669 | 0.646 |
| deny | 50 | 10 | 0.524 | 0.434 | 0.571 | 0.582 |
| yield | 50 | 10 | 0.500 | 0.489 | 0.588 | 0.429 |
[ Back to the navigation ] [ Back to the content ]