[ Skip to the content ]

Institute of Formal and Applied Linguistics Wiki

[ Back to the navigation ]

You are here: start » courses » rg » 2012 » encouraging-consistent-translation-bushra

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
courses:rg:2012:encouraging-consistent-translation-bushra [2012/10/23 15:28]
jawaid 		courses:rg:2012:encouraging-consistent-translation-bushra [2012/10/29 20:30] (current)
jawaid
Line 3: Line 3:
  
 ====Related Work:==== ====Related Work:====
-- Paper on the similar approach by Carput (2009) has found to be different in comparison to this work. They have used "one translation per discourse" approach as a post-processing step for MT whereas this works integrate the proposed scheme inside the decoding model of MT system. The later approach could also affect the selection of neighboring phrases whereas same doesn't hold for Carput (2009). +  - Paper on the similar approach by Carput (2009) has found to be different in comparison to this work. They have used "one translation per discourse" approach as a post-processing step for MT whereas this work integrates the proposed scheme inside the decoding model of MT system. The later approach could also affect the selection of neighboring phrases whereas same doesn't hold for Carput (2009). 
- +  - Without giving any proper evidence, authors have speculated that modeling "one-sense-per-discourse" is similar to the human translators' nature of making consistent word choices for specific domains. 
-- Without giving any proper evidence, authors have speculated that modeling "one-sense-per-discourse" is similar to the human translators' nature of making consistent word choices for specific domains. +
  
 ====Analysis:==== ====Analysis:====
-* Forced Decoding is the decoding method in which for a given pair of source and target sentences, decoder searches for the translation rules that fit the target sentence for a given source sentence. +  - Forced Decoding is the decoding method in which for a given pair of source and target sentences, decoder searches for the translation rules that fit the target sentence for a given source sentence. 
- +  - Term "cases" represent situations in which multiple occurrences of a given source phrase (f) in a document (d) might be translated using more than one different translation rules resulting in different target translations. Such cases are shown in Table 1, numbers shown in column "Translation counts" are the number of sentences in which source phrase occurred. E.g. in 4th row, Korea=2 shows that word "Korea" is produced as target translation in 2 sentences whereas in last column same source phrase is translated as "the" in one sentence and as "which" in other sentence. 
-2- Term "cases" represent situations in which multiple occurrences of a given source phrase (f) in a document (d) might be translated using more than one different translation rules resulting in different target translations. Such cases are shown in Table 1, numbers shown in column "Translation counts" are the number of sentences in which source phrase occurred. E.g. in 4th row, Korea=2 shows that word "Korea" is produced as target translation in 2 sentences whereas in last column same source phrase is translated as "the" in one sentence and as "which" in other sentence. +  - After selecting sample cases, few filtering techniques have been applied to discard the irrelevant samples. Filtering steps are well documented in the paper on page 419 in first paragraph of 2nd column.
- +
-3- After selecting sample cases, few filtering techniques have applied to discard the irrelevant samples. Filtering steps are well documented in the paper on page 419 in first paragraph of 2nd column. +
- +
-Approach: +
- +
-1- The core idea of maintaining translation consistencies (TC) is implemented by intrdocuing bias towards TC in form of "consistency features". Three consistency features are used inside the decoding model and their values are estimated using 2-pass decoding scheme. +
- +
-2- BM25 which is used as term weighting function is a well known ranking function in the filed of information retrieval and a refined version of TF-IDF (another ranking function uses in IR). +
- +
-3- C1 is a fine-grain approach of term weighting function and it is computed by counting how many times rule was applied in first-pass. This approach suffers when source and target phrase differs only in non-terminal positioning or use of determiners. +
- +
-4- C2 on other hand is a coarse-grain function which takes only target tokens into account. To us, C2 looks similar to the language model feature but trained only on the target side of the dev set. +
- +
-5- C3 goes over all alignment pairs and for each rule it select those term pairs that have maximum feature vaue. +
- +
-Evaluation: +
- +
-1- Cdec's implementation of Hierarchical MT is used in this work. As we know, hierarchical decoding is also implemented in other MT systems such as Moses, Joshua etc. The selection of cdec over other MT systems is authors' personal choice and doest not bring extra benefits. +
- +
-2- MIRA is used to train the MT system. +
- +
-3- Authors don't tune decoder in first-pass i.e. they don't calulcate feature weights (lambda) and probably they use weights from their previous experiments or setups. They don't clearly state the reason of this decision but our hypothesis is they might skiped the tuning step just to speed up the translation process. +
- +
-4- NIST-BLEU is used to compare results with official NIST evaluation whereas IBM-BLEU is used for evaluating the rest of experiments. We don't fully understand the use of different BLEU (prefering shorter sentences incase of NIST and longer incase of IBM) for evaluation and not sticking with NIST-BLEU only. +
- +
-5- They gain maximum of 1.0 point increase in BLEU after combining all three features. +
- +
-6- Authors called BLEU as a "conservative measure" due to negatively marking their system whenever a selection of content word didn't have exact match in reference translation. We strongly disagree with their claim because baseline is also evaluated using same the metric and it also suffers with decreasy in accuracy because of content word mismatch. We see two solutions to the mentioned issue:+
  
- i- They could have supported their argument by manually evaluating the test set. +====Approach:==== 
- ii- Instead of wasting half of the page length by criticizing over BLEU, they could have evaluated their system on other metric such as METEOR.+  - The core idea of maintaining translation consistencies (TC) is implemented by introducing bias towards TC in the form of "consistency features". Three consistency features are used inside the decoding model and their values are estimated using 2-pass decoding scheme. 
 +  - BM25 which is used as term weighting function is a well known ranking function in the filed of information retrieval and a refined version of TF-IDF (another ranking function used in IR). 
 +  - Description of consistency features: 
 +    - C<sub>1</sub> is a fine-grain approach of term weighting function and it is computed by counting how many times rule was applied in first-pass. This approach suffers when source and target phrase differs only in non-terminal positioning or with the presence of determiners. 
 +    - C<sub>2</sub> on other hand is a coarse-grain function which takes only target tokens into account. To us, C<sub>2</sub> looks similar to the language model feature but trained only on the target side of the dev set. 
 +    - C<sub>3</sub> goes over all alignment pairs and for each rule it selects those term pairs that have maximum feature value.
  
-7- Also, significance testing is not performed.+====Evaluation:====
  
-Conclusion:+  - Cdec's implementation of Hierarchical MT is used in this work. As we know, hierarchical decoding is also implemented in other MT systems such as Moses, Joshua etc. The selection of cdec over other MT systems is authors' personal choice and doest not bring extra benefits. 
 +  - MIRA is used for tuning feature weights. 
 +  - Authors don't tune decoder in first-pass i.e. they don't calculate feature weights (lambda) and probably they use weights from their previous experiments or setups. They don't clearly state the reason of this decision but our hypothesis is they might skipped the tuning step just to speed up the translation process. 
 +  - NIST-BLEU (prefers shorter sentences) is used to compare results with official NIST evaluation whereas IBM-BLEU (prefers longer sentences) is used for evaluating the rest of experiments. We don't fully understand the use of different BLEU for evaluation and why they didn't use only NIST-BLEU for evaluation. (MP: that's not exact, see [[courses:rg:2012:encouraging-consistent-translation]]) 
 +  - They gain maximum of 1.0 point increase in BLEU after combining all three features. 
 +  - Authors called BLEU as a "conservative measure" due to negatively marking their system whenever a selection of content word didn't have exact match in reference translation. We strongly disagree with their claim because baseline is also evaluated using same the metric and it also suffers with decreasy in accuracy because of content word mismatch. We see two solutions to the mentioned issue: 
 +    - They could have supported their argument by manually evaluating the test set. 
 +    - Instead of wasting half of the page length by criticizing over BLEU, they could have evaluated their system on other metric such as METEOR. 
 +  - We believe that significance testing should have been performed.
  
-Paper is nicely written and all experiments are well documented. We believe that consistent translation choices system is better suited only for translating from direction of morphologically-rich to morphologically-low language pairs but in case of reverse direction this approach can make serious errors by putting different morhological forms of the words bearing different meanings under the conistent translations.+====Conclusion:==== 
 +Paper is nicely written and all experiments are well documented. We believe that consistent translation choices system is well suited only for translating from direction of morphologically-rich to morphologically-low language pairs and not the other way round. For translating in direction of morphologically rich languages, this approach can make serious errors by putting different morphological forms of the words, bearing different meanings, under the consistent translations.

[ Back to the navigation ] [ Back to the content ]