[ Skip to the content ]

Institute of Formal and Applied Linguistics Wiki

[ Back to the navigation ]

You are here: start » courses » rg » 2012 » encouraging-consistent-translation

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
courses:rg:2012:encouraging-consistent-translation [2012/10/17 11:43]
dusek 		courses:rg:2012:encouraging-consistent-translation [2012/10/23 11:04] (current)
popel my remarks
Line 8: Line 8:
 The list of discussed topics follows the outline of the paper: The list of discussed topics follows the outline of the paper:
 ==== Sec. 2. Related Work ==== ==== Sec. 2. Related Work ====
- 
 **Differences from Carpuat 2009** **Differences from Carpuat 2009**
   * It is different: the decoder just gets additional features, but the decision is up to it -- Carpuat 2009 just post-edits the outputs and substitutes the most likely variant everywhere   * It is different: the decoder just gets additional features, but the decision is up to it -- Carpuat 2009 just post-edits the outputs and substitutes the most likely variant everywhere
Line 17: Line 16:
     * The authors do not state their evidence clearly.     * The authors do not state their evidence clearly.
     * One sense is not the same as one translation     * One sense is not the same as one translation
-==== Sec. 3. Exploratory analysis ==== 
  
 +==== Sec. 3. Exploratory analysis ====
 **Hiero** **Hiero**
   * The idea would most probably work the same in normal phrase-based SMT, but the authors use hierarchical phrase-based translation (Hiero)   * The idea would most probably work the same in normal phrase-based SMT, but the authors use hierarchical phrase-based translation (Hiero)
Line 39: Line 38:
  
 ==== Sec. 4. Approach ==== ==== Sec. 4. Approach ====
- 
 The actual experiments begin only now; the used data is different. The actual experiments begin only now; the used data is different.
  
 **Choice of features** **Choice of features**
-  * They define 3 features that are designed to be biased towrds consistency -- or are they?+  * They define 3 features that are designed to be biased towards consistency -- or are they?
     * If e.g. two variants are used 2 times each, they will have roughly the same score     * If e.g. two variants are used 2 times each, they will have roughly the same score
   * The BM25 function is a refined version of the [[http://en.wikipedia.org/wiki/TF-IDF|TF-IDF]] score   * The BM25 function is a refined version of the [[http://en.wikipedia.org/wiki/TF-IDF|TF-IDF]] score
Line 59: Line 57:
     * but rules are very similar, so we also need something less fine-grained     * but rules are very similar, so we also need something less fine-grained
   * C2 is a target-side feature, just counts the target side tokens (only the "most important" ones; in terms of TF-IDF)   * C2 is a target-side feature, just counts the target side tokens (only the "most important" ones; in terms of TF-IDF)
-    * It may be compared to Language Model features, but is trained only on the target part of the bilingual training data.+    * It may be compared to Language Model features, but is trained only on the target part of the bilingual tuning data.
   * C3 counts occurrences of source-target token pairs (and uses the "most important" term pair for each rule, again)   * C3 counts occurrences of source-target token pairs (and uses the "most important" term pair for each rule, again)
  
Line 65: Line 63:
   * They need two passes through the data   * They need two passes through the data
   * You need to have document segmentation   * You need to have document segmentation
-    * Since the frequencies are trained on the training set, you can just translate one document at a time, no need to have full sets of documents+    * Since the frequencies are trained on the tuning set (see Sec. 5), you can just translate one document at a time, no need to have full sets of documents 
 + 
 +==== Sec. 5. Evaluation and Discussion ==== 
 +**Choice of baseline** 
 +  * Baselines are quite nice and competitive, we believe this really is an improvement 
 +  * MIRA is very cutting-edge 
 + 
 +**Tuning the feature weights** 
 +  * For the 1st phase, "heuristically" probably means they just used some reasonable enough values, e.g. from earlier experiments 
 +    * This is in order to speed up the experiment, they don't want to wait for MIRA twice. 
 + 
 +**The usage of two variants of the BLEU evaluation metric** 
 +  * The BLEU variants do not differ that much, only in Brevity Penalty for multiple references 
 +    * IBM BLEU uses the reference that is closest to the MT output (in terms of length), NIST BLEU uses the shortest one 
 +  * This was probably just due to some technical reasons, e.g. they had their optimization software designed for one metric and not the other 
 + 
 +**Performance** 
 +  * Adding any single feature improves the performance, the combination of all of them is even better 
 +    * "Most works in MT just achieve 1 BLEU point improvement, then it is ready to be published" :-) 
 +  * There are no significance tests !! 
 +    * Even 1.0 BLEU doesn't have to be significant if there is a lot of changes 
 + 
 +**Selection of the sentences for the analysis** 
 +  * They only select cases with bigger differences due to filtering -- this leads to skewed selection of sentences where BLEU changes more 
 +    * This can lead to an improvement in "big" things (such as selection of content words), but a worsening in "small" things (grammatical words, inflection) 
 + 
 +**BLEU deficiency** 
 +  * The authors argue that some of the sentences are not worsened, but since the changed words do not appear in the reference, BLEU scoring hurts their system 
 +  * We believe this argument is misleading: 
 +    * The baseline has the same problem 
 +    * The human translators use different expression in the reference for a reason (even if the meaning is roughly the same, there can be style differences etc.) 
 +    * We must be careful when we criticize BLEU -- it is all too easy to find single sentences where it failed 
 +      * It's always better to back up our argument by human rankings 
 +    * Why didn't they run METEOR or other metric and left it for future work? 
 + 
 +==== Sec. 6. Conclusions ==== 
 +**Structural variation moderation** 
 +  * Sounds a bit sci-fi, but very interesting 
 + 
 +**Choice of discourse context** 
 +  * It's true that choosing just document works well for news articles, but not for most of the content we wish to translate 
 +  * Domain feature, topic modelling or word classes should be worth trying 
 + 
 +===== Our conclusion ===== 
 + 
 +Nice paper with a very good idea that probably can improve translations, but with several arguments that are not backed up by sufficient evidence or clearly misleading. The initial analysis is done in a very precise and detailed way. The actual translation experiments show that adding new features helps, but lack some obvious steps, such as significance checking or actually proving that BLEU is wrong by using METEOR or similar metric. 
 + 
 +===== Martin's remarks ===== 
 +  * The approach (without modifications) does not seem to be suitable for translating to a morphologically rich language. Different forms of the same lemma would be considered different senses (if not grouped together due to 1/2 of character being same), so the system would produce e.g. only nominatives. 
 +  * Also, there should be a modification for source-side words with more possible PoS. E.g. "book" as a noun should be translated differently than as a verb and you can easily find both in one document.

[ Back to the navigation ] [ Back to the content ]