Both sides previous revision
Previous revision
Next revision
|
Previous revision
Next revision
Both sides next revision
|
external:addicter [2011/05/16 07:05] mphi |
external:addicter [2011/05/16 07:47] mphi |
| |
==== Word Alignment -- Progress and Results ==== | ==== Word Alignment -- Progress and Results ==== |
| |
| Latest best results: [[http://mtj.ut.ee/addicter-best.txt|txt]] |
| |
=== Alternative model comparison === | === Alternative model comparison === |
|| || Precision/Recall/F-score: |||||||| | | ^ Precision/Recall/F-score: ^^^^ |
|| || Lex || Order || Punct || Miss || | | ^ Lex ^ Order ^ Punct ^ Miss ^ |
|| meteor ||0.092/0.251/**0.135** ||0.047/0.229/**0.078** ||0.248/0.665/**0.361** ||0.020/0.382/**0.038** || | ^ meteor |0.092/0.251/**0.135** |0.047/0.229/**0.078** |0.248/0.665/**0.361** |0.020/0.382/**0.038** | |
|| ter* ||0.106/0.387/**0.167** ||0.025/0.191/**0.044** ||0.132/0.936/**0.232** ||0.026/0.170/**0.046** || | ^ ter* |0.106/0.387/**0.167** |0.025/0.191/**0.044** |0.132/0.936/**0.232** |0.026/0.170/**0.046** | |
|| hmm ||0.162/0.426/**0.234** ||0.069/0.309/**0.112** ||0.281/0.793/**0.415** ||0.025/0.400/**0.047** || | ^ hmm |0.162/0.426/**0.234** |0.069/0.309/**0.112** |0.281/0.793/**0.415** |0.025/0.400/**0.047** | |
|| lcs ||0.168/0.462/**0.247** ||0.000/0.000/**0.000** ||0.293/0.848/**0.435** ||0.026/0.374/**0.049** || | ^ lcs |0.168/0.462/**0.247** |0.000/0.000/**0.000** |0.293/0.848/**0.435** |0.026/0.374/**0.049** | |
|| gizadiag* ||0.183/0.512/**0.270** ||0.044/0.250/**0.075** ||0.285/0.784/**0.417** ||0.038/0.224/**0.065** || | ^ gizainter |0.170/0.483/**0.252** |0.049/0.137/**0.072** |0.284/0.878/**0.429** |0.029/0.409/**0.054** | |
|| gizainter ||0.170/0.483/**0.252** ||0.049/0.137/**0.072** ||0.284/0.878/**0.429** ||0.029/0.409/**0.054** || | ^ gizadiag* |0.183/0.512/**0.270** |0.044/0.250/**0.075** |0.285/0.784/**0.417** |0.038/0.224/**0.065** | |
|| berkeley* ||0.200/0.540/**0.291** ||0.050/0.330/**0.087** ||0.292/0.844/**0.434** ||0.039/0.267/**0.068** || | ^ berkeley* |0.200/0.540/**0.291** |0.050/0.330/**0.087** |0.292/0.844/**0.434** |0.039/0.267/**0.068** | |
| |
=== Explicit wrong lex choice detection === | === Explicit wrong lex choice detection === |
| * align input+czeng to reference+czeng and input+czeng to hypotheses+czeng |
| * extract hypothesis-to-reference alignments from there |
| |
* align input+czeng to reference+czeng and input+czeng to hypotheses+czeng | | ^ Precision/Recall/F-score: ^^^^ |
* extract hypothesis-to-reference alignments from there | | ^ Lex ^ Order ^ Punct ^ Miss ^ |
| ^ czengdiag* |0.187/0.514/**0.275** |0.069/0.455/**0.120** |0.230/0.883/**0.365** |0.035/0.234/**0.061** | |
|| border=1 | ^ czenginter |0.197/0.543/**0.290** |0.108/0.475/**0.176** |0.233/0.926/**0.372** |0.032/0.402/**0.060** | |
|| || Precision/Recall/F-score: |||||||| | |
|| || Lex || Order || Punct || Miss || | |
|| czengdiag* ||0.187/0.514/**0.275** ||0.069/0.455/**0.120** ||0.230/0.883/**0.365** ||0.035/0.234/**0.061** || | |
|| czenginter ||0.197/0.543/**0.290** ||0.108/0.475/**0.176** ||0.233/0.926/**0.372** ||0.032/0.402/**0.060** || | |
| |
=== Alignment combinations === | === Alignment combinations === |
| |
via weighed HMM | via weighed HMM |
| |
|| border=1 | | ^ Precision/Recall/F-score: ^^^^ |
|| || Precision/Recall/F-score: |||||||| | | ^ Lex ^ Order ^ Punct ^ Miss ^ |
|| || Lex || Order || Punct || Miss || | ^ meteor+hmm |0.162/0.426/**0.234** |0.068/0.309/**0.112** |0.286/0.794/**0.421** |0.025/0.400/**0.047** | |
|| meteor+hmm ||0.162/0.426/**0.234** ||0.068/0.309/**0.112** ||0.286/0.794/**0.421** ||0.025/0.400/**0.047** || | ^ ter+hmm |0.116/0.402/**0.180** |0.030/0.184/**0.051** |0.145/0.912/**0.251** |0.026/0.181/**0.046** | |
|| ter+hmm ||0.116/0.402/**0.180** ||0.030/0.184/**0.051** ||0.145/0.912/**0.251** ||0.026/0.181/**0.046** || | ^ gizadiag+hmm |0.186/0.515/**0.273** |0.040/0.215/**0.067** |0.297/0.836/**0.438** |0.039/0.238/**0.067** | |
|| gizadiag+hmm ||0.186/0.515/**0.273** ||0.040/0.215/**0.067** ||0.297/0.836/**0.438** ||0.039/0.238/**0.067**|| | ^ gizainter+hmm |0.194/0.505/**0.281** |0.062/0.282/**0.101** |0.299/0.806/**0.436** |0.033/0.382/**0.061** | |
|| gizainter+hmm ||0.194/0.505/**0.281** ||0.062/0.282/**0.101** ||0.299/0.806/**0.436** ||0.033/0.382/**0.061**|| | ^ berkeley+hmm |0.203/0.548/**0.297** |0.049/0.320/**0.085** |0.290/0.816/**0.428** |0.041/0.277/**0.071** | |
|| berkeley+hmm ||0.203/0.548/**0.297** ||0.049/0.320/**0.085** ||0.290/0.816/**0.428** ||0.041/0.277/**0.071**|| | ^ czengdiag+hmm |0.190/0.517/**0.278** |0.073/0.457/**0.126** |0.291/0.841/**0.432** |0.039/0.238/**0.067** | |
|| czengdiag+hmm ||0.190/0.517/**0.278** ||0.073/0.457/**0.126** ||0.291/0.841/**0.432** ||0.039/0.238/**0.067**|| | ^ czenginter+hmm |0.214/0.545/**0.307** |0.093/0.525/**0.158** |0.304/0.818/**0.443** |0.038/0.363/**0.068** | |
|| czenginter+hmm ||0.214/0.545/**0.307** ||0.093/0.525/**0.158** ||0.304/0.818/**0.443** ||0.038/0.363/**0.068** || | | ||||| |
|| |||||||||| | ^ berkeley+czenginter+hmm |0.219/0.568/**0.316** |0.070/0.432/**0.120** |0.298/0.817/**0.436**|0.048/0.290/**0.082** | |
|| berkeley+czenginter+hmm ||0.219/0.568/**0.316** ||0.070/0.432/**0.120** ||0.298/0.817/**0.436**||0.048/0.290/**0.082** || | ^ berkeley+czenginter+gizainter+hmm |0.220/0.569/**0.317** |0.068/0.420/**0.118** |0.298/0.812/**0.436**|0.048/0.290/**0.083** | |
|| berkeley+czenginter+gizainter+hmm ||0.220/0.569/**0.317** ||0.068/0.420/**0.118** ||0.298/0.812/**0.436**||0.048/0.290/**0.083** || | ^ berkeley+czenginter+meteor+hmm |0.220/0.569/**0.317** |0.070/0.440/**0.121** |0.295/0.810/**0.433**|0.048/0.290/**0.083** | |
|| berkeley+czenginter+meteor+hmm ||0.220/0.569/**0.317** ||0.070/0.440/**0.121** ||0.295/0.810/**0.433**||0.048/0.290/**0.083** || | ^ berkeley+czenginter+meteor+gizainter+hmm |0.221/0.571/**0.318** |0.068/0.424/**0.118** |0.298/0.808/**0.436** |0.049/0.292/**0.084** | |
| |
==== TODOs ==== | ==== TODOs ==== |
* try domain adaptation for word alignment, EMNLP 2011 paper | * test alignment with synonym detection (cz_wn required) = separating ''lex::'' and ''disam::'' |
* test alignment with synonym detection (cz_wn required) = separating @@lex@@ and @@disam@@ | |
* order evaluation | * order evaluation |
* a lot of background research | * a lot of background research |
* if it (and only it) were to be moved in the original permutation, what would be the best place? | * if it (and only it) were to be moved in the original permutation, what would be the best place? |
* evaluate with nr. of intersections | * evaluate with nr. of intersections |
| * try domain adaptation for word alignment, EMNLP 2011 paper |
* comb and comment the code | * comb and comment the code |
* add help files | * add help files |
* integrate with the rest of Addicter | * integrate with the rest of Addicter |
* learner's corpus | * approach applicable to learner's corpora |
* see Anne Lüdelig, TLT9 | * see Anne Lüdelig, TLT9 |
* adapt to Sara's program | * adapt Addicter to Sara's program |
* alternative to reference-based evaluation: "Inconsistencies in Penn parsing", M. Dickinson | * alternative to reference-based evaluation: "Inconsistencies in Penn parsing", M. Dickinson |
| |