Both sides previous revision
Previous revision
Next revision
|
Previous revision
Next revision
Both sides next revision
|
courses:rg:wishlist [2012/10/22 22:28] machacek |
courses:rg:wishlist [2012/10/29 14:58] bilek |
==== MT Evaluation ==== | ==== MT Evaluation ==== |
* Martin Popel would appreciate two RG meetings devoted to significance tests & MT evaluation. The two presenters should together read the following 4 papers (and related ones) and select two for presenting (one on bootstrap, one on approximate randomization). | * Martin Popel would appreciate two RG meetings devoted to significance tests & MT evaluation. The two presenters should together read the following 4 papers (and related ones) and select two for presenting (one on bootstrap, one on approximate randomization). |
- Stefan Riezler and John T. Maxwell III: [[http://acl.ldc.upenn.edu/W/W05/W05-0908.pdf|On Some Pitfalls in Automatic Evaluation and Significance Testing for MT]] (page 67) ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation, 2005. | - <del>Stefan Riezler and John T. Maxwell III: [[http://acl.ldc.upenn.edu/W/W05/W05-0908.pdf|On Some Pitfalls in Automatic Evaluation and Significance Testing for MT]] (page 67) ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation, 2005.</del> |
- Nicolas Stroppa, Karolina Owczarzak, Andy Way: [[http://doras.dcu.ie/15227/1/stroppa_owczarzak_07.pdf|A Cluster-Based Representation for Multi-System MT Evaluation]], 2007. | - Nicolas Stroppa, Karolina Owczarzak, Andy Way: [[http://doras.dcu.ie/15227/1/stroppa_owczarzak_07.pdf|A Cluster-Based Representation for Multi-System MT Evaluation]], 2007. |
- <del>Philipp Koehn: [[http://acl.ldc.upenn.edu/acl2004/emnlp/pdf/Koehn.pdf|Statistical significance tests for machine translation evaluation]], EMNLP 2004.</del> | - <del>Philipp Koehn: [[http://acl.ldc.upenn.edu/acl2004/emnlp/pdf/Koehn.pdf|Statistical significance tests for machine translation evaluation]], EMNLP 2004.</del> |
- Ying Zhang, Stephan Vogel, Alex Waibel: [[http://www.lrec-conf.org/proceedings/lrec2004/pdf/755.pdf|Interpreting BLEU/NIST Scores: How Much Improvement Do We Need to Have a Better System?]] | - Ying Zhang, Stephan Vogel, Alex Waibel: [[http://www.lrec-conf.org/proceedings/lrec2004/pdf/755.pdf|Interpreting BLEU/NIST Scores: How Much Improvement Do We Need to Have a Better System?]] |
| * T. Berg-Kirkpatrick, D. Burkett, D. Klein: [[http://www.aclweb.org/anthology/D/D12/D12-1091.pdf|An Empirical Investigation of Statistical Significance in NLP]] |
| |
* Chi-kiu LO and Dekai WU: [[http://www.cs.ust.hk/~dekai/library/WU_Dekai/LoWu_Acl2011.pdf|MEANT: An inexpensive, high-accuracy, semi-automatic metric for evaluating translation utility via semantic frames. ACL HLT 2011]] (or other MEANT or HMEANT paper, but this one seems to be THE main one) | * Chi-kiu LO and Dekai WU: [[http://www.cs.ust.hk/~dekai/library/WU_Dekai/LoWu_Acl2011.pdf|MEANT: An inexpensive, high-accuracy, semi-automatic metric for evaluating translation utility via semantic frames. ACL HLT 2011]] (or other MEANT or HMEANT paper, but this one seems to be THE main one) |