[ Skip to the content ]

Institute of Formal and Applied Linguistics Wiki

• ÚFAL Wiki Home
• ÚFAL Home
• Sitemap
• RSS
• ÚFAL calendar
• Recent changes
• Login
• Hledat:

[ Back to the navigation ]

Differences

This shows you the differences between two versions of the page.

View differences: Side by SideInline Go

Link to this comparison view

Both sides previous revision Previous revision 2009/03/11 14:58 hladka 2009/03/11 14:58 hladka 2009/03/11 14:57 hladka 2009/03/10 10:18 hladka 2009/03/10 10:18 hladka 2009/03/10 10:12 hladka 2009/03/10 09:36 hladka 2009/03/09 10:42 hladka 2009/03/09 10:41 hladka 2009/03/09 10:41 hladka 2009/03/09 10:40 hladka 2009/03/09 10:37 hladka 2009/03/02 10:01 hladka 2009/03/02 09:54 hladka 2009/03/02 09:54 hladka 2009/03/02 09:52 hladka 2009/03/02 08:40 mirovsky 2009/03/02 08:38 mirovsky 2009/03/02 08:37 mirovsky 2009/03/01 17:42 hladka 2009/03/01 17:42 hladka 2009/03/01 17:26 hladka 2009/03/01 17:24 hladka 2009/03/01 17:24 hladka 2009/03/01 17:23 hladka 2009/03/01 17:23 hladka 2009/03/01 17:21 hladka 2009/02/26 13:19 hladka 2009/02/26 13:18 hladka 2009/02/26 13:12 hladka 2009/02/26 12:50 hladka 2009/02/26 12:49 hladka 2009/02/26 12:45 hladka 2009/02/26 12:40 hladka 2009/02/26 12:16 hladka 2009/02/26 12:15 hladka 2009/02/26 12:15 hladka 2009/02/26 12:14 hladka 2009/02/26 12:14 hladka 2009/02/26 12:13 hladka 2009/02/26 12:11 hladka 2009/02/26 12:11 hladka 2009/02/26 12:03 hladka 2009/02/26 12:02 hladka 2009/02/26 12:01 hladka 2009/02/26 11:57 hladka 2009/02/26 11:56 hladka 2009/02/26 11:52 hladka 2009/02/26 11:49 hladka 2009/02/26 09:54 mirovsky 2009/02/26 09:43 mirovsky Go Next revision		Previous revision 2009/03/25 09:04 mirovsky 2009/03/24 21:55 hladka 2009/03/24 21:09 hladka 2009/03/17 08:18 hladka 2009/03/17 08:14 hladka 2009/03/16 11:15 hladka 2009/03/16 11:14 hladka 2009/03/16 11:13 hladka 2009/03/11 15:10 hladka 2009/03/11 15:00 hladka 2009/03/11 14:59 hladka 2009/03/11 14:58 hladka 2009/03/11 14:58 hladka 2009/03/11 14:57 hladka 2009/03/10 10:18 hladka 2009/03/10 10:18 hladka 2009/03/10 10:12 hladka 2009/03/10 09:36 hladka 2009/03/09 10:42 hladka 2009/03/09 10:41 hladka 2009/03/09 10:41 hladka 2009/03/09 10:40 hladka 2009/03/09 10:37 hladka 2009/03/02 10:01 hladka 2009/03/02 09:54 hladka 2009/03/02 09:54 hladka 2009/03/02 09:52 hladka 2009/03/02 08:40 mirovsky 2009/03/02 08:38 mirovsky 2009/03/02 08:37 mirovsky 2009/03/01 17:42 hladka 2009/03/01 17:42 hladka 2009/03/01 17:26 hladka 2009/03/01 17:24 hladka 2009/03/01 17:24 hladka 2009/03/01 17:23 hladka 2009/03/01 17:23 hladka 2009/03/01 17:21 hladka 2009/02/26 13:19 hladka 2009/02/26 13:18 hladka 2009/02/26 13:12 hladka 2009/02/26 12:50 hladka 2009/02/26 12:49 hladka 2009/02/26 12:45 hladka 2009/02/26 12:40 hladka 2009/02/26 12:16 hladka 2009/02/26 12:15 hladka 2009/02/26 12:15 hladka 2009/02/26 12:14 hladka 2009/02/26 12:14 hladka 2009/02/26 12:13 hladka Go Next revision Both sides next revision
user:hladka:playcoref [2009/03/01 17:23] hladka		user:hladka:playcoref [2009/03/02 09:54] hladka
Line 136:		Line 136:
	**BH**: Jirka ma pravdu. Pocitani skore musi byt objektivni. Proto jsem vzorecek upravila tak, ze nebude pocitat shodu hrace vzhledem k rucni anotaci.		**BH**: Jirka ma pravdu. Pocitani skore musi byt objektivni. Proto jsem vzorecek upravila tak, ze nebude pocitat shodu hrace vzhledem k rucni anotaci.
-
-
-
-
	===== Output Data Needed =====		===== Output Data Needed =====
	* score list ## //player_id, pts, #sessions//		* score list ## //player_id, pts, #sessions//
-	* documents after the ''n''-th session consist of ''2*n'' players coreference annotation (some of them should be identical, the more identical the better); how to calculate an inter-player agreement? **BH:** v clanku, ktery budeme posilat na ACL, by mela byt seriozni uvaha o kvalite dat, ktere ziskame z her. Kvalita jde ruku v ruce s mezihracskou shodou a shodou mezi hracem a automatickou procedurou. **Pavle**, vzal by sis prosim tuto cast na starosti? Jiz jsem si prochazela nejake prace a zatim mi z toho vychazi, ze je vhodne okomentovat:	+	* documents after the ''n''-th session consist of ''2*n'' players coreference annotation (some of them should be identical, the more identical the better); how to calculate an inter-player agreement? **BH:** v clanku, ktery budeme posilat na ACL, by mela byt seriozni uvaha o kvalite dat, ktera ziskame z her. Kvalita jde ruku v ruce s mezihracskou shodou a shodou mezi hracem a automatickou procedurou. **Pavle**, vzal by sis prosim tuto cast na starosti? Jiz jsem prochazela nejake prace a zatim mi z toho vychazi, ze je vhodne okomentovat:
-	- kappa measure	+	(**JM**: Mluvil jsem kvůli měření mezianotátorské shody v anotování koreference se Zdeňkem a vyšlo z toho, že na měření shody na šipkách by použil prostě jen F-measure. Její smysl je jasný a je symetrická. Kappa je nevhodná kvůli tomu, že pravděpodobnost náhodné shody je poměrně nízká a těžko se určuje; kappa se hodí spíš pro klasifikační úlohy (proto ji použiju v Anjiině projektu na shodu v určování typu koreference, když už se shodli na šipce); ostatní (G-theory a Pearson correlation) neznám, jsem zvědav, co k tomu řekne Pavel.)
-	- G-theory	+	- kappa measure
-	- the Pearson correlation - viz (Snow et al., 2008) [[http://ufal.mff.cuni.cz/~hladka/gwap/amt_emnlp08_accepted.pdf|Cheap and Fast - But is it Good? ... ]],	+	- G-theory - see [[http://en.wikipedia.org/wiki/Generalizability_theory|wiki]], [[http://www.aclweb.org/anthology-new/J/J07/J07-1002.pdf|Petra Saskia Bayerl; Karsten Ingmar Paul
		+	Identifying Sources of Disagreement: Generalizability Theory in Manual Annotation Studies]], Computational Linguistics, Volume 33, Number 1, March 2007.
		+	- the Pearson correlation - see (Snow et al., 2008) [[http://ufal.mff.cuni.cz/~hladka/gwap/amt_emnlp08_accepted.pdf|Cheap and Fast - But is it Good? ... ]]
	* session		* session
	* player_A_id, player_B_id		* player_A_id, player_B_id
Line 153:		Line 151:
	* number of corrections by player_A and by player_B (**JM**: I do not see the point in this)		* number of corrections by player_A and by player_B (**JM**: I do not see the point in this)
	* corrections by player_A and by player_B (**JM**: and maybe nor in this) (**BH**: I am interested in the manner of the players. Maybe the corrections will be total mess, but we have to see the data at least from the very first sessions. )		* corrections by player_A and by player_B (**JM**: and maybe nor in this) (**BH**: I am interested in the manner of the players. Maybe the corrections will be total mess, but we have to see the data at least from the very first sessions. )
		+
	===== Design =====		===== Design =====

---

[ Back to the navigation ] [ Back to the content ]