| Both sides previous revision
Previous revision
Next revision
|
Previous revision
Next revision
Both sides next revision
|
user:hladka:playcoref [2009/02/26 13:12]
hladka |
user:hladka:playcoref [2009/03/02 09:52]
hladka |
| |
| **BH**: Jirka ma pravdu. Pocitani skore musi byt objektivni. Proto jsem vzorecek upravila tak, ze nebude pocitat shodu hrace vzhledem k rucni anotaci. | **BH**: Jirka ma pravdu. Pocitani skore musi byt objektivni. Proto jsem vzorecek upravila tak, ze nebude pocitat shodu hrace vzhledem k rucni anotaci. |
| |
| |
| |
| |
| ===== Output Data Needed ===== | ===== Output Data Needed ===== |
| * score list ## //player_id, pts, #sessions// | * score list ## //player_id, pts, #sessions// |
| * documents after the ''n''-th session consist of ''2*n'' players coreference annotation (some of them should be identical, the more identical the better); how to calculate an inter-player agreement? | * documents after the ''n''-th session consist of ''2*n'' players coreference annotation (some of them should be identical, the more identical the better); how to calculate an inter-player agreement? **BH:** v clanku, ktery budeme posilat na ACL, by mela byt seriozni uvaha o kvalite dat, ktera ziskame z her. Kvalita jde ruku v ruce s mezihracskou shodou a shodou mezi hracem a automatickou procedurou. **Pavle**, vzal by sis prosim tuto cast na starosti? Jiz jsem prochazela nejake prace a zatim mi z toho vychazi, ze je vhodne okomentovat: |
| | (**JM**: Mluvil jsem kvůli měření mezianotátorské shody v anotování koreference se Zdeňkem a vyšlo z toho, že na měření shody na šipkách by použil prostě jen F-measure. Její smysl je jasný a je symetrická. Kappa je nevhodná kvůli tomu, že pravděpodobnost náhodné shody je poměrně nízká a těžko se určuje; kappa se hodí spíš pro klasifikační úlohy (proto ji použiju v Anjiině projektu na shodu v určování typu koreference, když už se shodli na šipce); ostatní (G-theory a Pearson correlation) neznám, jsem zvědav, co k tomu řekne Pavel.) |
| | - kappa measure |
| | - G-theory - see [[http://en.wikipedia.org/wiki/Generalizability_theory|wiki]], [[http://www.aclweb.org/anthology-new/J/J07/J07-1002.pdf|Petra Saskia Bayerl; Karsten Ingmar Paul |
| | Identifying Sources of Disagreement: Generalizability Theory in Manual Annotation Studies]], Computational Linguistics, Volume 33, Number 1, March 2007. |
| | - the Pearson correlation - see (Snow et al., 2008) [[http://ufal.mff.cuni.cz/~hladka/gwap/amt_emnlp08_accepted.pdf|Cheap and Fast - But is it Good? ... ]] |
| * session | * session |
| * player_A_id, player_B_id | * player_A_id, player_B_id |
| * Linh's coreference resolution procedure **---PS TO DO---** What type of input data the Linh's procedure works with? ''tool_chain'' is going to be extended by the ''S'' option enabling to run Vasek Klimes' t-parser in a basic version, i.e. just t-tree and functors. See more info [[https://wiki.ufal.ms.mff.cuni.cz/user:hladka:playcoref#automaticke-urcovani-koreference-v-ceskych-datech-prehled]]. | * Linh's coreference resolution procedure **---PS TO DO---** What type of input data the Linh's procedure works with? ''tool_chain'' is going to be extended by the ''S'' option enabling to run Vasek Klimes' t-parser in a basic version, i.e. just t-tree and functors. See more info [[https://wiki.ufal.ms.mff.cuni.cz/user:hladka:playcoref#automaticke-urcovani-koreference-v-ceskych-datech-prehled]]. |
| * conversion: csts <-> pml m_coref scheme | * conversion: csts <-> pml m_coref scheme |
| | |
| | |
| |
| |
| ====== ACL - IJCNLP2009 ====== | ====== ACL - IJCNLP2009 ====== |
| * [[http://www.acl-ijcnlp-2009.org/|Suntec Singapore, August 2-7, 2009]] | * [[http://www.acl-ijcnlp-2009.org/|Suntec Singapore, August 2-7, 2009]] |
| * [[http://www.acl-ijcnlp-2009.org/main/callforpapers.html#shortpapers|Short papers]], deadline: April 26, 2009 | * [[http://www.acl-ijcnlp-2009.org/main/callforpapers.html#shortpapers|Short papers]], deadline: April 26, 2009. Predposledni verze clanku musi byt hotova do 12. dubna. Nasledne clanek posleme vybranym kolegum, aby meli na precteni a okomentovani tyden. Nam pak bude zbyvat tyden do terminu. |
| * pracovni adresar ''/net/work/projects/playlang/doc/ACL-IJCNLP2009/'' | * pracovni adresar ''/net/work/projects/playlang/doc/ACL-IJCNLP2009/'' |
