IPSJ SIG Technical Report Vol.2014-NL-219 No /12/17 1,a) Graham Neubig 1,b) Sakriani Sakti 1,c) 1,d) 1,e) 1. [23] 1(a) 1(b) [19] n-best [1] 1 N

Size: px

Start display at page:

Download "IPSJ SIG Technical Report Vol.2014-NL-219 No /12/17 1,a) Graham Neubig 1,b) Sakriani Sakti 1,c) 1,d) 1,e) 1. [23] 1(a) 1(b) [19] n-best [1] 1 N"

れんかはにうだ
4 years ago
Views:

1 1,a) Graham Neubig 1,b) Sakriani Sakti 1,c) 1,d) 1,e) 1. [23] 1(a) 1(b) [19] n-best [1] 1 Nara Institute of Science and Technology a) akabe.koichi.zx8@is.naist.jp b) neubig@is.naist.jp c) ssakti@is.naist.jp d) tomoki@is.naist.jp e) s-nakamura@is.naist.jp 1 In general, the zen sect was knowledge, and it is valued enlightenment. In general, the zen sect was knowledge, and it is valued enlightenment. "values" reord +"instead of" (a) In general, the zen sect was knowledge, and it is valued enlightenment. was, and it Insertion err "instead of" Reordering err is valued "values" (b) [24] n-gram 2 n-gram [1] 1

2 [9] [5], [10] 2. n-gram n-gram [1], [24] n-gram 1(b) n-gram ( 1 ) ( 2 ) n-gram n-gram n-gram n-gram ( 3 ) n-gram n-gram n-gram n-gram n-gram ID n-gram n-gram [24] 4 n-gram n-gram 1-best n-gram [6] n-gram I like information technology! hate not information technology Ref I do n't like IT! 2 institute of technology n-gram add-one n-gram n-gram [7] 1-best n-gram n-best BLEU+1[12] n-gram n-gram 3. 1-best n-gram [24] n-gram n-gram n-gram [4] [17] n-gram 2 I don t like IT! I like information technology! like information 2

3 Ins L-Del R-Del A D B C A B C (a) Ins (b) Del Rep Reord Reord Reord A C E B A C D B E A B C D A B C D E (c) Rep (d) Reord 3 n-gram like information n-gram n-gram BLEU+1 BLEU+1 n-gram METEOR[3] METEOR 1-best 1-best 4. 2 n-gram enshou enshou e e P E f the central figure around that time was enshou fuketsu. enshou a (1)-(1,2) (2)-(3) (3)-(4) -(5) (4)-(6) (5)-(8) (6)-(7) (7)-(9) t err a 1-1,2 : a 5-8 : a 6-7 : a -5 : p 4.5 : p 5.5 : p 6.5 : t p 1:a 1-1,2 3.5: a -5 5: a 5-8 6: a : p : p : p 6.5 [9] [5], [10] 2 ( 1 ) ( 2 ) e e P E a a 3

4 e P E a e e P E 2 KFTT Train 330k 5.91M 6.09M Dev k 26.8k Test k 28.5k 4.2 n-gram e e P E a n-gram t 1 ( 1 ) 1 e P E e e e P E ( 2 ) e p {, 1, 1.5, 2, 2.5, }.5 0 ( 3 ) e p p e p ( 1 ) e p {1, 2, 3, } 3(a) (c) ( 2 ) e P E e p {, 1.5, 2.5, } 3(b) ( 3 ) e p e e P E p {, 1.5, 2.5, } 3(d) 4.3 n-gram n-gram n-gram T F A P R P = T T + F, R = T A (1) T n-gram 1 F 2 F ( 1 ) n-gram ( 2 ) n-gram 4 (a) 2-gram 2 F 2 (b) 3-gram 2 T 2 F 1 (c) T (KFTT)[14] 2 Travatar [15] Forest-to-String MERT [16] BLEU[18] 2 4 4

5 n-gram n-gram n-gram (a) True: 0 False: 2 (b) True: 2 False: 1 (c) True: 4 False: 0 4 n-gram Threshold = No paraphrase n-best BLEU+1[12] 3.1.0[13] METEOR version 1.5 [8] METEOR n-best 100 L1 [22] KFTT 0015 n-gram 1-gram 3-gram KFTT Dev LM n-gram n-gram n-gram 0.1 n-gram n-gram

6 the members of the kanoha group were... and castles as the shogunate s official painters......, rinzai school started with its founder gigen rinzai at the end of the... kaishou LM (w/ BLEU+1) LM (w/ METEOR) Found errors w/o Paraphrase w/ Paraphrase Insertion Deletion Replacement Reordering All 7 8 n-gram 3 1 n-gram 2 n-gram 5.4 METEOR 7 BLEU+1 METEOR n-gram BLEU Dev BLEU BLEU n-best 1-best

7 JSPS QE: Quality Estimation [2] QE BLEU+1 METEOR TER[21] RIBES[11] [1], [20], [24] 7. BLEU METEOR [1] Akabe, K., Neubig, G., Sakti, S., Toda, T. and Nakamura, S.: Discriminative Language Models as a Tool for Machine Translation Error Analysis, Proc. COLING, pp (2014). [2] Bach, N., Huang, F. and Al-Onaizan, Y.: Goodness: A Method for Measuring Machine Translation Confidence, Proc. ACL, pp (2011). [3] Banerjee, S. and Lavie, A.: METEOR: An automatic metric for evaluation with improved correlation with human judgments, Proc. ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization (2005). [4] Bannard, C. and Callison-Burch, C.: Paraphrasing with bilingual parallel corpora, Proc. ACL, pp (2005). [5] Berka, J., Bojar, O., Fishel, M., Popovic, M. and Zeman, D.: Automatic Error Analysis: Hjerson Helping Addicter, Proc. LREC (2012). [6] Church, K. W. and Hank, P.: Word association norms, mutual information, and lexicography, Computational Linguistics, pp (1990). [7] Collins, M.: Discriminative Training Methods for Hidden Markov Models: Theory and Experiments with Perceptron Algorithms, Proc. EMNLP, pp. 1 8 (2002). [8] Denkowski, M. and Lavie, A.: Meteor Universal: Language Specific Translation Evaluation for Any Target Language, Proceedings of the EACL 2014 Workshop on Statistical Machine Translation (2014). [9] Fishel, M., Bojar, O. and Popović, M.: Terra: a Collection of Translation Error-Annotated Corpora., Proc. LREC, pp (2012). [10] Fishel, M., Bojar, O., Zeman, D. and Berka, J.: Automatic translation error analysis, Text, Speech and Dialogue, Springer, pp (2011). [11] Isozaki, H., Hirao, T., Duh, K., Sudoh, K. and Tsukada, H.: Automatic Evaluation of Translation Quality for Distant Language Pairs, Proc. EMNLP, pp (2010). [12] Lin, C.-Y. and Och, F. J.: Orange: a method for evaluating automatic evaluation metrics for machine translation, Proc. COLING, pp (2004). [13] Mizukami, M., Neubig, G., Sakti, S., Toda, T. and Nakamura, S.: Building a Free, General-Domain Paraphrase Database for Japanese, Proc. COCOSDA (2014). [14] Neubig, G.: The Kyoto Free Translation Task, http: // (2011). [15] Neubig, G.: Travatar: A Forest-to-String Machine Translation Engine based on Tree Transducers, Proc. ACL Demo Track, pp (2013). [16] Och, F. J.: Minimum Error Rate Training in Statistical Machine Translation, Proc. ACL, pp (2003). [17] Onishi, T., Utiyama, M. and Sumita, E.: Paraphrase Lattice for Statistical Machine Translation, Proc. ACL, pp. 1 5 (2010). [18] Papineni, K., Roukos, S., Ward, T. and Zhu, W.-J.: 7

8 BLEU: a method for automatic evaluation of machine translation, Proc. ACL, pp (2002). [19] Popović, M.: Hjerson: An open source tool for automatic error classification of machine translation output, The Prague Bulletin of Mathematical Linguistics, Vol. 96, No. 1, pp (2011). [20] Popović, M. and Ney, H.: Towards automatic error analysis of machine translation output, Computational Linguistics, Vol. 37, No. 4, pp (2011). [21] Snover, M., Dorr, B., Schwartz, R., Micciulla, L. and Makhoul, J.: A study of translation edit rate with targeted human annotation, Proc. AA, pp (2006). [22] Tibshirani, R.: Regression shrinkage and selection via the lasso, Journal of the Royal Statistical Society, pp (1996). [23] Vilar, D., Xu, J., d Haro, L. F. and Ney, H.: Error analysis of statistical machine translation output, Proc. LREC, pp (2006). [24] Neubig, G. Sakti, S. 216 (SIG-NL) (2014). 8

9 (LM) n-gram n-gram (LM) 0.1 [1] [24] n-gram n-gram Threshold = No paraphrase Threshold = No paraphrase

10 LM (w/ BLEU+1) LM (w/ METEOR) LM (w/ BLEU+1) LM (w/ METEOR) w/o Paraphrase w/o Paraphrase Found errors w/ Paraphrase Found errors w/ Paraphrase Insertion Deletion Replacement Reordering All Insertion Deletion Replacement Reordering All

2

NTT 2012 NTT Corporation. All rights reserved. 2 3 4 5 Noisy Channel f : (source), e : (target) ê = argmax e p(e f) = argmax e p(f e)p(e) 6 p( f e) (Brown+ 1990) f1 f2 f3 f4 f5 f6 f7 He is a high school