[ Skip to the content ]

Institute of Formal and Applied Linguistics Wiki

[ Back to the navigation ]

You are here: start » courses » rg » 2012 » sigtest-mt-zilka

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision Both sides next revision
courses:rg:2012:sigtest-mt-zilka [2012/11/14 17:45]
zilka 		courses:rg:2012:sigtest-mt-zilka [2012/11/14 18:02]
zilka
Line 47: Line 47:
 ===== Section 4, 5 ===== ===== Section 4, 5 =====
   * we cannot use Student's T distribution to estimate confidence interval for BLEU, because it cannot be constructed in the form of sum of terms to give us mean and variance   * we cannot use Student's T distribution to estimate confidence interval for BLEU, because it cannot be constructed in the form of sum of terms to give us mean and variance
-  * so for estimating the confidence intervals we will use randomized test set generation - e.g. we build 1000 new test sets of size 300 sentences out of our small test set of 300 sentences (i.e. we draw (with replacement) samples from the small test set; so we should get 1000 different test sets)+  * so for estimating the confidence intervals we will use randomized test set generation (= bootstrap resampling) - e.g. we build 1000 new test sets of size 300 sentences out of our small test set of 300 sentences (i.e. we draw (with replacement) samples from the small test set; so we should get 1000 different test sets)
   * answer to Question3 - they do not assume there is any particular distribution in the set of BLEU scores of the 1000 test sets (i.e. their method would work regardless of whether the distribution is normal, uniform or any other), but it is perhaps normally distributed   * answer to Question3 - they do not assume there is any particular distribution in the set of BLEU scores of the 1000 test sets (i.e. their method would work regardless of whether the distribution is normal, uniform or any other), but it is perhaps normally distributed
  
 ===== Section 6 ===== ===== Section 6 =====
 +  * they use bootstrap resampling to compare 2 systems; we want to determine whether system 1 is better than system 2; we want to determine that from a set of differences of system's performances (i.e. difference of score of system 1 and score of system 2)
 +    * so we determine in what percent of cases system 1 beats system 2, and that's our final confidence that system 1 is better than system 2 (e.g. 45 times out of 50 -> 90% confidence)
 +  * the rest of the paper just proves that the assumption is correct
  
 +===== Martin's explanation of p-values =====
 +  * two philosophical views of p-value - Fisher's and Person's - unfortunately their are mixed in modern textbooks which only confuses us
 +  * we always set a null hypothesis H0 as: systems are the same, and alternative hypothesis HA: there is difference in the systems; P(H0) + P(HA) = 1
 +  * p-value =
 +    * P(T(X)>=T(x_orig)|H0) = P(x|H0) = //if the compared systems are the same, what's the probability that we see this data//
 +    * unfortunately we tend to view the p-value as P(H0|x) which it is not and we need to apply the Bayes's theorem to get it
 +  * bootstrap resampling can be viewed as p-value=P(d(x) > d(x_orig)|H0), and is approximated by S/B; where S is number of system 1 beating system 2 and B is number of measurements
  
- 
- 
-  * **Section 3** describes the data 

[ Back to the navigation ] [ Back to the content ]