BLEU Kishore Papineni, Salim Roukos, Todd Ward and Wei-Jing Zhu. (2002) BLEU: a method for Automatic Evaluation of Machine Translation. ACL. MT ( ) MT

Size: px

Start display at page:

Download "BLEU Kishore Papineni, Salim Roukos, Todd Ward and Wei-Jing Zhu. (2002) BLEU: a method for Automatic Evaluation of Machine Translation. ACL. MT ( ) MT"

せとかおうじ
7 years ago
Views:

1 4. 1

2 BLEU Kishore Papineni, Salim Roukos, Todd Ward and Wei-Jing Zhu. (2002) BLEU: a method for Automatic Evaluation of Machine Translation. ACL. MT ( ) MT ( ) BLEU 2

3 BLEU BLEU MT NTCIR-7 3

4 ngram 1 MT ngram MT 1. 1 It is a guide to action 2 which 3 ensures that the military 4always obyes the 5 commands 6 of the party. 2. It is to ensure the troops forever hearing the activity guidebook that party direct 1. 1 It is a guide to action that 3 ensures that the military will forever heed Party 5 commands. 2. It is the guiding principle 2 which guarantees the military forces 4 always being under the command 6 of the party. 3. It is the practical guide for the army 4 always to heed the directions 6 of the party. 4

5 1. ngram MT ngram 2. 5

6 ngram ngram = ngram MT ngram MT: the the the the the the the Ref1: The cat is on the mat. Ref2: There is a cat on the mat. MT the the Ref1 Ref2 1gram = 7 7 6

7 ngram ngram ngram P n = MT ngram MT: the the the the the the the Ref1: The cat is on the mat. Ref2: There is a cat on the mat. P 1 = 2 7 P 2 = 0 7

8 MT1: P 1 = = 0.94, P 2 = = 0.59 MT2: P 1 = 8 14 = 0.57, P 2 = 1 13 = 0.08 MT 1. 1 It is a guide to action 2 which 3 ensures that the military 4always obyes the 5 commands 6 of the party. 2. It is to ensure the troops forever hearing the activity guidebook that party direct 1. 1 It is a guide to action that 3 ensures that the military will forever heed Party 5 commands. 2. It is the guiding principle 2 which guarantees the military forces 4 always being under the command 6 of the party. 3. It is the practical guide for the army 4 always to heed the directions 6 of the party. 8

9 ngram P n = MT MT ngram ngram MT MT ngram ngram MT ngram MT ngram MT 9

10 ngram N 1 n=1 N log P n P n = ngram N = ngram ( 4 ) ngram 10

11 ngram P n MT ngram ngram MT P n MT ngram P 1 = 2/2, P 2 = 1/1 MT : of the 1: It is a guide to action that ensures that the military will forever heed Party commands. 2: It is the guiding principle which guarantees the military forces always being under the command of the party. 3: It is the practical guide for the army always to heed the directions of the party. 11

12 MT MT ( ) c = MT MT r = MT BP (brevity penalty) BP = 1 if c r exp(1 r/c) if c < r MT BP = 1 ( ) MT < BP < 1 12

13 BLEU BLEU = BP exp( N 1 n=1 N log P n) MT ngram BLEU BLEU... MT 13

14 BLEU 10 ( ) 2 10 ( ) (S1,S2,S3,H1,H2) H1 H2 H1 H2 1( ) 5( )

15 (paired t-test) u i s r(u, i, s) s r(u, i, s ) s s d(u, i, s, s ) = r(u, i, s) r(u, i, s ) m(s, s ) = u,i d(u, i, s, s ) v(s, s ) = 1 n u,i (d(u, i, s, s ) m(s, s )) 2 n = u,i 1. s s m(s, s ) = 0 0 t = m(s, s ) v(s,s ) n 1 t 15

16 BLEU BLEU 2 S S S H H S1 0 S2 m(s1, S2) S3 = S2+m(S3, S2) 16

17 BLEU Normalized score Bilingual Monolingual BLEU S1 S2 S3 H1 H2 0-1 BLEU 2 BLEU S3 H1 H1 H2 S2 S3 17

18 BLEU Monolingual Judgment monolingual.txt f(x) BLEU BLEU 18

19 BLEU Bilingual Judgment bilingual.txt f(x) BLEU BLEU 2 19

20 BLEU MT BLEU BLEU ngram 20

Rapp BLEU[10] [9] BLEU OrthoBLEU Rapp OrthoBLEU [9] OrthoBLEU OrthoBLEU ) ) ) 1) NTT Natural Language Research

Rapp BLEU[10] [9] BLEU OrthoBLEU Rapp OrthoBLEU [9] OrthoBLEU OrthoBLEU ) ) ) 1) NTT Natural Language Research RJ-008 Is Back Translation Really Unuseful? Validation of Back Translation from the Perspective of a Checking Method for Users Mai Miyabe Takashi Yoshino 1. [1, 2] [3] [4] 1 2 2 [3,5,6,7] [8, 9] 1: 2 3