4. BLEU @NICT mutiyama@nict.go.jp 1
BLEU Kishore Papineni, Salim Roukos, Todd Ward and Wei-Jing Zhu. (2002) BLEU: a method for Automatic Evaluation of Machine Translation. ACL. MT ( ) MT ( ) BLEU 2
BLEU BLEU MT NTCIR-7 3
ngram 1 MT ngram MT 1. 1 It is a guide to action 2 which 3 ensures that the military 4always obyes the 5 commands 6 of the party. 2. It is to ensure the troops forever hearing the activity guidebook that party direct 1. 1 It is a guide to action that 3 ensures that the military will forever heed Party 5 commands. 2. It is the guiding principle 2 which guarantees the military forces 4 always being under the command 6 of the party. 3. It is the practical guide for the army 4 always to heed the directions 6 of the party. 4
1. ngram MT 2. 1. ngram 2. 5
ngram ngram = ngram MT ngram MT: the the the the the the the Ref1: The cat is on the mat. Ref2: There is a cat on the mat. MT the the Ref1 Ref2 1gram = 7 7 6
ngram ngram ngram P n = MT ngram MT: the the the the the the the Ref1: The cat is on the mat. Ref2: There is a cat on the mat. P 1 = 2 7 P 2 = 0 7
MT1: P 1 = 17 18 = 0.94, P 2 = 10 17 = 0.59 MT2: P 1 = 8 14 = 0.57, P 2 = 1 13 = 0.08 MT 1. 1 It is a guide to action 2 which 3 ensures that the military 4always obyes the 5 commands 6 of the party. 2. It is to ensure the troops forever hearing the activity guidebook that party direct 1. 1 It is a guide to action that 3 ensures that the military will forever heed Party 5 commands. 2. It is the guiding principle 2 which guarantees the military forces 4 always being under the command 6 of the party. 3. It is the practical guide for the army 4 always to heed the directions 6 of the party. 8
ngram P n = MT MT ngram ngram MT MT ngram ngram MT ngram MT ngram MT 9
ngram N 1 n=1 N log P n P n = ngram N = ngram ( 4 ) ngram 10
ngram P n MT ngram ngram MT P n MT ngram P 1 = 2/2, P 2 = 1/1 MT : of the 1: It is a guide to action that ensures that the military will forever heed Party commands. 2: It is the guiding principle which guarantees the military forces always being under the command of the party. 3: It is the practical guide for the army always to heed the directions of the party. 11
MT MT ( ) c = MT MT r = MT BP (brevity penalty) BP = 1 if c r exp(1 r/c) if c < r MT BP = 1 ( ) MT < BP < 1 12
BLEU BLEU = BP exp( N 1 n=1 N log P n) MT ngram BLEU BLEU... MT 13
BLEU 10 ( ) 2 10 ( ) 500 50 5 (S1,S2,S3,H1,H2) H1 H2 H1 H2 1( ) 5( ) 2 0 14
(paired t-test) u i s r(u, i, s) s r(u, i, s ) s s d(u, i, s, s ) = r(u, i, s) r(u, i, s ) m(s, s ) = u,i d(u, i, s, s ) v(s, s ) = 1 n u,i (d(u, i, s, s ) m(s, s )) 2 n = u,i 1. s s m(s, s ) = 0 0 t = m(s, s ) v(s,s ) n 1 t 15
BLEU BLEU 2 S1 0.0527 0 0 S2 0.0829 0.326 0.551 S3 0.0930 0.44 0.691 H1 0.1934 2.265 2.574 H2 0.2571 2.8 2.612 2 S1 0 S2 m(s1, S2) S3 = S2+m(S3, S2) 16
BLEU 1 0.8 Normalized score 0.6 0.4 0.2 0 Bilingual Monolingual BLEU S1 S2 S3 H1 H2 0-1 BLEU 2 BLEU S3 H1 H1 H2 S2 S3 17
BLEU 3 2.5 2 Monolingual Judgment 1.5 1 0.5 0 monolingual.txt f(x) -0.5 0 0.05 0.1 0.15 0.2 0.25 0.3 BLEU BLEU 18
BLEU 3 2.5 2 Bilingual Judgment 1.5 1 0.5 0 bilingual.txt f(x) -0.5 0 0.05 0.1 0.15 0.2 0.25 0.3 BLEU BLEU 2 19
BLEU MT BLEU BLEU ngram 20