自然言語処理22_289

Size: px

Start display at page:

Download "自然言語処理22_289"

とよみこびき
4 years ago
Views:

1 (RNN) Dyer (Dyer, Clark, Lavie, and Smith 2011) IBM (Brown, Pietra, Pietra, and Mercer 1993) word embedding word embedding RNN (Yang, Liu, Li, Zhou, and Yu 2013) IBM 4 Recurrent Neural Networks for Word Alignment Akihiro Tamura, Taro Watanabe and Eiichiro Sumita This paper proposes a novel word alignment model based on a recurrent neural network (RNN), in which an unlimited alignment history is represented by recurrently connected hidden layers. In addition, we perform unsupervised learning inspired by (Dyer et al. 2011), which utilizes artificially generated negative samples. Our alignment model is directional, like the generative IBM models (Brown et al. 1993). To overcome this limitation, we encourage an agreement between the two directional models by introducing a penalty function, which ensures word embedding consistency across two directional models during training. The RNN-based model outperforms both the feed-forward NN-based model (Yang et al. 2013) and the IBM Model 4 under Japanese-English and French-English word alignment tasks, and achieves comparable translation performance to those baselines under Japanese-English and Chinese- English translation tasks., National Institute of Information and Communications Technology, Google

2 Vol. 22 No. 4 December 2015 Key Words: Word Alignment, Recurrent Neural Network, Unsupervised Learning, Agreement Constraint 1 IBM 1-5 (Brown et al. 1993) HMM (Vogel, Ney, and Tillmann 1996) (Och and Ney 2003; Berg-Kirkpatrick, Bouchard-Côté, DeNero, and Klein 2010) Yang (FFNN) Context-Dependent Deep Neural Network for HMM (CD-DNN-HMM) (Dahl, Yu, Deng, and Acero 2012) HMM IBM 4 HMM (Yang et al. 2013) FFNN-HMM (NN) (RNN) RNN RNN FFNN (Mikolov, Karafiát, Burget, Cernocký, and Khudanpur 2010; Mikolov and Zweig 2012; Sundermeyer, Oparin, Gauvain, Freiberg, Schlüter, and Ney 2013) (Auli, Galley, Quirk, and Zweig 2013; Kalchbrenner and Blunsom 2013) RNN RNN FFNN RNN FFNN NN Yang IBM HMM 290

3 (Yang et al. 2013) RNN Dyer (Dyer et al. 2011) RNN f e e f (Matusov, Zens, and Ney 2004; Liang, Taskar, and Klein 2006; Graça, Ganchev, and Taskar 2008; Ganchev, Graça, and Taskar 2008) f e e f 2 RNN word embedding word embedding RNN FFNN IBM 4 FFNN IBM 4 1 FFNN F1 IBM F1 FFNN 0.74% (BLEU) IBM % (BLEU) RNN 2 3 RNN 1 NN IBM 4 291

4 Vol. 22 No. 4 December RNN (Brown et al. 1993; Vogel et al. 1996; Och and Ney 2003) (Taskar, Lacoste-Julien, and Klein 2005; Moore 2005; Blunsom and Cohn 2006) FFNN (Yang et al. 2013) 2.1 J f J 1 = f 1,..., f J I e I 1 = e 1,..., e I f J 1 e I 1 a J 1 = a 1,..., a J a j f j e aj null (e 0 ) f j a j = 0 f J 1 e I 1 e I 1 p(f J 1 e I 1) = a J 1 p(f J 1, a J 1 e I 1). (1) IBM 1 2 HMM (1) a J 1 p(f J 1, a J 1 e I 1) p a p t 2 p(f J 1, a J 1 e I 1) = J p a (a j a j 1, j)p t (f j e aj ). (2) j=1 3 HMM p a (a j a j a j 1 ) (fertility) (distortion) IBM 3-5 EM (Dempster, Laird, and Rubin 1977) 2 p a a 0 =0 292

5 (f J 1, e I 1) (3) â J 1 â J 1 = argmax p(f1 J, a J 1 e I 1). (3) a J 1 HMM (Viterbi 1967) 2.2 FFNN FFNN (Dahl et al. 2012) (Le, Allauzen, and Yvon 2012; Vaswani, Zhao, Fossum, and Chiang 2013) (Collobert and Weston 2008; Collobert, Weston, Bottou, Karlen, Kavukcuoglu, and Kuksa 2011) Yang FFNN CD-DNN-HMM (Dahl et al. 2012) HMM (Yang et al. 2013) FFNN FFNN (2) p a p t FFNN J s NN (a J 1 f1 J, e I 1) = t a (a j a j 1 c(e aj 1 )) t t (f j, e aj c(f j ), c(e aj )). (4) j=1 t a t t p a p t s NN a J 1 c( w) w HMM a j 1 FFNN 1 t t (f j, e aj c(f j ), c(e aj )) lookup 1 L {H, B H } {O, B O } L word embedding (Bengio, Ducharme, Vincent, and Janvin 2003) V f V e word embedding M L M ( V f + V e ) V f V e unk null null 293

6 Vol. 22 No. 4 December FFNN t t(f j, e aj c(f j), c(e aj )) f j e aj 1 3 lookup L word embedding z 0 lookup z 0 z 0 z 1 3 z 1 = f(h z 0 + B H ), (5) t t = O z 1 + B O. (6) H B H O B O z 1 z 0 z z f(x) (Yang et al. 2013) htanh(x) 4 t a (a j a j 1 c(e aj 1 )) (7) (SGD) 5 (Rumelhart, Hinton, and Williams 1986) loss(θ) = (f,e) T max{0, 1 s θ (a + f, e) + s θ (a f, e)}. (7) 3 NN FFNN RNN 1 l z l = f(h l z l 1 + B Hl ) 4 x < 1 htanh(x) = 1 x > 1 htanh(x) = 1 htanh(x) = x 5 RNN SGD D SGD 294

7 θ T s θ θ a J 1 (4) a + a θ 3 RNN a J 1 RNN s NN (a J 1 f J 1, e I 1) = J j=1 t RNN (a j a j 1 1, f j, e aj ). (8) t RNN a j FFNN a j 1 j 1 a j 1 1 FFNN 2 RNN lookup L {H d, R d, B d H } {O, B O} H d R d B d H a j 1 d (d = a j a j 1 ) d {H 8, H 7,, H 7, H 8, R 8, R 7,, R 7, R 8, B 8 H, B 7 H,, B7 H, B 8 H } y j FFNN 2 f 1 f J a j y i a 1 a j 1 2 RNN 295

8 Vol. 22 No. 4 December f j e aj f j e aj 2 lookup lookup 2 word embedding word embedding x j lookup FFNN lookup lookup x j j 1 y j 1 H d R d B d H a j 1 d y j j + 1 y j f j e aj t RNN (a j a j 1 1, f j, e aj ) 6 y j = f(h d x j + R d y j 1 + B d H), (9) t RNN = O y j + B O. (10) H d R d B d H O B O y j x j y j y j 1 y j 1 1 y j 1 1 y j 1 = y j f(x) (Yang et al. 2013) htanh(x) FFNN 2 RNN d 1 RNN y i FFNN 4 L H d R d B H d O B O SGD D = 1 SGD (Rumelhart et al. 1986) j 6 j = 1 a 0 0 y 0 y 1 = f(h d x 1 + B d H ). 296

9 FFNN (7) Dyer contrastive estimation (CE) (Smith and Eisner 2005) (Dyer et al. 2011) CE Dyer T V e T V e CE { loss(θ) = max 0, 1 (f +,e + ) T E Φ [s θ (a f +, e + )] + (f +,e ) Ω } E Φ [s θ (a f +, e )]. (11) Φ (f, e) E Φ [s θ ] Φ s θ Ω T V e e + T e e + (e V e+ e ) (11) Ω Noise Contrastive Estimation (Gutmann and Hyvärinen 2010; Mnih and Teh 2012) Negative Sampling (Mikolov, Sutskever, Chen, Corrado, and Dean 2013) f + e e + N W loss(θ) = { max 0, 1 E GEN [s θ (a f +, e + )] + 1 } E GEN [s θ (a f +, e )]. (12) N f + T e (12) e + f + ((f +, e + ) T ) e e + e + = e 297

10 Vol. 22 No. 4 December 2015 N f + GEN Φ e e + e V e l 0 IBM 1 (Vaswani, Huang, and Chiang 2012) f i f + C l 0 IBM 1 IBM 1 e 4.2 FFNN RNN f e f j e 1 Yang FFNN (Yang et al. 2013) Matusov Liang f e e f 2 2 (Matusov et al. 2004; Liang et al. 2006) Ganchev Graça EM E (Ganchev et al. 2008; Graça et al. 2008) word embedding word embedding { argmin loss(θf E ) + loss(θ EF ) + α θ LEF θ LF E }. (13) θ F E, θ EF θ F E θ EF f e e f θ L lookup L word embedding α θ θ 2- (13) loss(θ) (7) (12)

11 Algorithm 1 : θf 1 E, θef 1, T, MaxIter, D, N, C, IBM 1, W, α 1: for all t such that 1 t MaxIter do 2: {(f +, e + ) D } sample(d, T ) 3-1: {(f +, {e } N ) D } neg e ({(f +, e + ) D }, N, C, IBM 1) 3-2: {(e +, {f } N ) D } neg f ({(f +, e + ) D }, N, C, IBM 1) 4-1: θ t+1 F E update({f +, e +, {e } N ) D, θfe t, θef t, W, α) 4-2: θ t+1 EF update({e+, f +, {f } N ) D, θef t, θfe t, W, α) 5: end for : θ MaxIter+1 EF, θ MaxIter+1 FE T D (f +, e + ) D f + e + l 0 IBM 1 (IBM1) N ({e } N {f } N ) SGD θ F E θ EF word embedding θ EF θ F E θ F E θ EF NAACL 2003 shared task (Mihalcea and Pedersen 2003) Hansards (Hansards) Basic Travel Expression Corpus (BTEC) (Takezawa, Sumita, Sugaya, Yamamoto, and Yamamoto 2002) (IWSLT a ) IWSLT2007 (Fordyce 2007) (IWSLT) FBIS (FBIS) NTCIR-9 NTCIR-10 (Goto, Lu, Chow, Sumita, and Tsou 2011; Goto, Chow, Lu, Sumita, and Tsou 2013) (NTCIR-9, NTCIR-10) 1 Train Dev Test IWSLT a IWSLT BTEC 7 t θef t θt F E t 1 θt 1 F E θt 1 EF θef t θt F E θt EF θt F E 299

12 Vol. 22 No. 4 December Train Dev Test Hansards 100 K BTEC IWSLT a 9 K IWSLT 40 K 2, FBIS NIST K 878 NIST04 1,597 NTCIR M 2,000 2,000 NTCIR M 2,000 2,000 IWSLT a IWSLT 9,960 (Goh, Watanabe, Yamamoto, and Sumita 2010) 9,960 9, IWSLT a Hansards Hansards IWSLT a 2 8 NAACL 2003 shared task 110 Hansards 10 FBIS NIST02 NIST03 NIST RNN IBM 4 FFNN HMM Vogel (Vogel et al. 1996) (HMM indep ) Vogel Liang (Liang et al. 2006) (HMM joint ) IBM 4 IBM 1-4 HMM (Och and Ney 2003) IBM 1 HMM IBM (1 5 H ) GIZA++ (IBM4) HMM indep 8 IWSLT a 2,000 2 IWSLT a 300

13 HMM joint BerkleyAligner 9 Liang IBM 1 HMM 5 (Liang et al. 2006) FFNN word embedding M 30 5 z = z FFNN (Yang et al. 2013) 2.2 FFNN s FFNN s+c FFNN u FFNN u+c s u +c RNN word embedding M 30 y j 100 x j 60 = 30 2 FFNN RNN s RNN s+c RNN u RNN u+c 4 FFNN RNN M 2 NN lookup L word embedding [ 0.1, 0.1] word embedding Mikolov (Mikolov et al. 2010) RNNLM 10 5 unk SGD D l2 0.1 W N C α 0.1 (SMT) Moses (Koehn, Hoang, Birch, Callison-Burch, Federico, Bertoldi, Cowan, Shen, Moran, Zens, Dyer, Bojar, Constrantin, and Herbst 2007) ChaSen 11 Stanford Chinese segmenter SRILM (Stolcke 2002) modified Kneser-Ney (Kneser and Ney 1995; Chen and Goodman 1996) IWSLT NTCIR-9 NTCIR-10 5 FBIS English Gigaword Xinhua

14 Vol. 22 No. 4 December SMT MERT (Och 2003) F1 NN (REF) IBM4 (IBM4) 2 Hansards REF f e e f grow-diag-final-and (Koehn, Och, and Marcu 2003) F1 5% FFNN FFNN s (REF/IBM4) ++ FFNN 2 IWSLT a Hansards HMM indep HMM joint IBM FFNN s (IBM4 ) FFNN s+c (IBM4 ) RNN s (IBM4 ) RNN s+c(ibm4 ) FFNN u FFNN u+c RNN u RNN u+c FFNN s (REF ) FFNN s+c (REF ) RNN s (REF ) RNN s+c (REF ) F1 302

15 FFNN s (REF/IBM4) IBM4 (REF) REF 2 IWSLT a Hansards RNN RNN u+c 2 IWSLT a Hansards RNN s/s+c/u/u+c (IBM4) RNN s/s+c (REF) FFNN s/s+c/u/u+c (IBM4) FFNN s/s+c (REF) IWSLT a RNN s (REF) RNN s (IBM4) FFNN s (REF) FFNN s (IBM4) RNN RNN Hansards RNN 6.1 IWSLT a Hansards RNN s+c (REF/IBM4) RNN u+c RNN s (REF/IBM4) RNN u FFNN s+c (REF/IBM4) FFNN u+c FFNN s (REF/IBM4) FFNN u FFNN RNN HMM joint HMM indep HMM Liang 6.2 IWSLT a RNN u RNN u+c RNN s (IBM4) RNN s+c (IBM4) Hansards FFNN IBM4 IBM4 FFNN RNN 303

16 Vol. 22 No. 4 December SMT BLEU4 13 (Papineni, Roukos, Ward, and Zhu 2002) MERT MERT 3 3 (Utiyama, Yamamoto, and Sumita 2009) IWSLT NTCIR-9 NTCIR-10 FBIS 10 IBM 4 SMT IBM4 all 5% (Koehn 2004) 3 * IBM4 FFNN s (IBM4) IBM4 all 2 3 (Yang et al. 2013) 3 RNN u RNN u+c FFNN s (IBM4) IBM4 NTCIR-9 FBIS IBM4 all IWSLT NTCIR-9 NTCIR-10 FBIS NIST03 NIST04 IBM4 all IBM FFNN s (IBM4 ) RNN s (IBM4 ) RNN s+c (IBM4 ) RNN u RNN u+c BLEU4 (%) 13 mteval-v13a.pl ( 304

17 6 6.1 RNN 3 FFNN s RNN s FFNN s RNN s 3 RNN s FFNN s 3(a) have you been FFNN s RNN s 5.3 RNN (IWSLT a ) (Hansards) (b) FFNN s RNN s 3 305

18 Vol. 22 No. 4 December BTEC 4 40 K 9 K 1 K IWSLT IWSLT a IWSLT a 1,000 IWSLT a 9 K 1 K 40 K (REF) 40 K 4 1 K RNN s+c (REF) 9 K RNN u+c 40 K IBM4 RNN IBM4 22.5% (9,000/40,000) 3 RNN u+c SMT IBM4 all SMT 4 HMM Liang Liang K 9 K 1 K HMM indep HMM joint IBM FFNN s (IBM4 ) FFNN s+c (IBM4 ) RNN s(ibm4 ) RNN s+c(ibm4 ) FFNN u FFNN u+c RNN u RNN u+c FFNN s (REF ) FFNN s+c (REF ) RNN s(ref ) RNN s+c(ref ) F1 306

19 RNN u+c RNN s+c (REF) RNN 7 RNN RNN Dyer (Dyer et al. 2011) word embedding RNN FFNN (Yang et al. 2013) (y i ) FFNN c(f j ) c(e aj ) Yang FFNN (Yang et al. 2013) Watanabe (Watanabe, Suzuki, Tsukada, and Isozaki 2006) SMT The 52nd Annual Meeting of the Association for Computational Linguistics (Tamura, Watanabe, and Sumita 2014) 307

20 Vol. 22 No. 4 December 2015 Auli, M., Galley, M., Quirk, C., and Zweig, G. (2013). Joint Language and Translation Modeling with Recurrent Neural Networks. In Proceedings of EMNLP 2013, pp Bengio, Y., Ducharme, R., Vincent, P., and Janvin, C. (2003). A Neural Probabilistic Language Model. Journal of Machine Learning Research, 3, pp Berg-Kirkpatrick, T., Bouchard-Côté, A., DeNero, J., and Klein, D. (2010). Painless Unsupervised Learning with Features. In Proceedings of HLT:NAACL 2010, pp Blunsom, P. and Cohn, T. (2006). Discriminative Word Alignment with Conditional Random Fields. In Proceedings of Coling/ACL 2006, pp Brown, P. F., Pietra, S. A. D., Pietra, V. J. D., and Mercer, R. L. (1993). The Mathematics of Statistical Machine Translation: Parameter Estimation. Computational Linguistics, 19 (2), pp Chen, S. F. and Goodman, J. (1996). An Empirical Study of Smoothing Techniques for Language Modeling. In Proceedings of ACL 1996, pp Collobert, R. and Weston, J. (2008). A Unified Architecture for Natural Language Processing: Deep Neural Networks with Multitask Learning. In Proceedings of ICML 2008, pp Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., and Kuksa, P. (2011). Natural Language Processing (Almost) from Scratch. Journal of Machine Learning Research, 12, pp Dahl, G. E., Yu, D., Deng, L., and Acero, A. (2012). Context-Dependent Pre-trained Deep Neural Networks for Large Vocabulary Speech Recognition. IEEE Transactions on Audio, Speech, and Language Processing, 20 (1), pp Dempster, A. P., Laird, N. M., and Rubin, D. B. (1977). Maximum Likelihood from Incomplete Data via the EM Algorithm. Journal of the Royal Statistical Society, Series B, 39 (1), pp Dyer, C., Clark, J., Lavie, A., and Smith, N. A. (2011). Unsupervised Word Alignment with Arbitrary Features. In Proceedings of ACL/HLT 2011, pp Fordyce, C. S. (2007). Overview of the IWSLT 2007 Evaluation Campaign. In Proceedings of IWSLT 2007, pp Ganchev, K., Graça, J. V., and Taskar, B. (2008). Better Alignments = Better Translations? In Proceedings of ACL/HLT 2008, pp Goh, C.-L., Watanabe, T., Yamamoto, H., and Sumita, E. (2010). Constraining a Generative Word Alignment Model with Discriminative Output. IEICE Transactions, 93-D (7), 308

21 pp Goto, I., Chow, K. P., Lu, B., Sumita, E., and Tsou, B. K. (2013). Overview of the Patent Machine Translation Task at the NTCIR-10 Workshop. In Proceedings of 10th NTCIR Conference, pp Goto, I., Lu, B., Chow, K. P., Sumita, E., and Tsou, B. K. (2011). Overview of the Patent Machine Translation Task at the NTCIR-9 Workshop. In Proceedings of 9th NTCIR Conference, pp Graça, J. V., Ganchev, K., and Taskar, B. (2008). Expectation Maximization and Posterior Constraints. In Proceedings of NIPS 2008, pp Gutmann, M. and Hyvärinen, A. (2010). Noise-Contrastive Estimation: A New Estimation Principle for Unnormalized Statistical Models. In Proceedings of AISTATS 2010, pp Kalchbrenner, N. and Blunsom, P. (2013). Recurrent Continuous Translation Models. In Proceedings of EMNLP 2013, pp Kneser, R. and Ney, H. (1995). Improved Backing-off for M-gram Language Modeling. In Proceedings of ICASSP 1995, pp Koehn, P. (2004). Statistical Significance Tests for Machine Translation Evaluation. In Proceedings of EMNLP 2004, pp Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W., Moran, C., Zens, R., Dyer, C., Bojar, O., Constrantin, A., and Herbst, E. (2007). Moses: Open Source Toolkit for Statistical Machine Translation. In Proceedings of ACL 2007, pp Koehn, P., Och, F. J., and Marcu, D. (2003). Statistical Phrase-Based Translation. In Proceedings of HLT/NAACL 2003, pp Le, H.-S., Allauzen, A., and Yvon, F. (2012). Continuous Space Translation Models with Neural Networks. In Proceedings of NAACL/HLT 2012, pp Liang, P., Taskar, B., and Klein, D. (2006). Alignment by Agreement. In Proceedings of HLT/NAACL 2006, pp Matusov, E., Zens, R., and Ney, H. (2004). Symmetric Word Alignments for Statistical Machine Translation. In Proceedings of Coling 2004, pp Mihalcea, R. and Pedersen, T. (2003). An Evaluation Exercise for Word Alignment. In Proceedings of the HLT-NAACL 2003 Workshop on Building and Using Parallel Texts: Data Driven Machine Translation and Beyond, pp Mikolov, T., Karafiát, M., Burget, L., Cernocký, J., and Khudanpur, S. (2010). Recurrent Neural Network based Language Model. In Proceedings of INTERSPEECH 2010, pp

22 Vol. 22 No. 4 December 2015 Mikolov, T., Sutskever, I., Chen, K., Corrado, G., and Dean, J. (2013). Distributed Representations of Words and Phrases and their Compositionality. In Proceedings of NIPS 2013, pp Mikolov, T. and Zweig, G. (2012). Context Dependent Recurrent Neural Network Language Model. In Proceedings of SLT 2012, pp Mnih, A. and Teh, Y. W. (2012). A Fast and Simple Algorithm for Training Neural Probabilistic Language Models. In Proceedings of ICML 2012, pp Moore, R. C. (2005). A Discriminative Framework for Bilingual Word Alignment. In Proceedings of HLT/EMNLP 2005, pp Och, F. J. (2003). Minimum Error Rate Training in Statistical Machine Translation. In Proceedings of ACL 2003, pp Och, F. J. and Ney, H. (2003). A Systematic Comparison of Various Statistical Alignment Models. Computational Linguistics, 29, pp Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002). BLEU: a Method for Automatic Evaluation of Machine Translation. In Proceedings of ACL 2002, pp Rumelhart, D. E., Hinton, G. E., and Williams, R. J. (1986). Learning Internal Representations by Error Propagation. In Rumelhart, D. E. and McClelland, J. L. (Eds.), Parallel Distributed Processing, pp MIT Press. Smith, N. A. and Eisner, J. (2005). Contrastive Estimation: Training Log-Linear Models on Unlabeled Data. In Proceedings of ACL 2005, pp Stolcke, A. (2002). SRILM - An Extensible Language Modeling Toolkit. In Proceedings of ICSLP 2002, pp Sundermeyer, M., Oparin, I., Gauvain, J.-L., Freiberg, B., Schlüter, R., and Ney, H. (2013). Comparison of Feedforward and Recurrent Neural Network Language Models. In Proceedings of ICASSP 2013, pp Takezawa, T., Sumita, E., Sugaya, F., Yamamoto, H., and Yamamoto, S. (2002). Toward a Broad-coverage Bilingual Corpus for Speech Translation of Travel Conversations in the Real World. In Proceedings of LREC 2002, pp Tamura, A., Watanabe, T., and Sumita, E. (2014). Recurrent Neural Networks for Word Alignment Model. In Proceedings of ACL 2014, pp Taskar, B., Lacoste-Julien, S., and Klein, D. (2005). A Discriminative Matching Approach to Word Alignment. In Proceedings of HLT/EMNLP 2005, pp Utiyama, M., Yamamoto, H., and Sumita, E. (2009). Two Methods for Stabilizing MERT: NICT at IWSLT In Proceedings of IWSLT 2009, pp

23 Vaswani, A., Huang, L., and Chiang, D. (2012). Smaller Alignment Models for Better Translations: Unsupervised Word Alignment with the l 0 -norm. In Proceedings of ACL 2012, pp Vaswani, A., Zhao, Y., Fossum, V., and Chiang, D. (2013). Decoding with Large-Scale Neural Language Models Improves Translation. In Proceedings of EMNLP 2013, pp Viterbi, A. J. (1967). Error Bounds for Convolutional Codes and an Asymptotically Optimum Decoding Algorithm. IEEE Transactions on Information Theory, 13 (2), pp Vogel, S., Ney, H., and Tillmann, C. (1996). Hmm-based Word Alignment in Statistical Translation. In Proceedings of Coling 1996, pp Watanabe, T., Suzuki, J., Tsukada, H., and Isozaki, H. (2006). NTT Statistical Machine Translation for IWSLT In Proceedings of IWSLT 2006, pp Yang, N., Liu, S., Li, M., Zhou, M., and Yu, N. (2013). Word Alignment Modeling with Context Dependent Deep Neural Network. In Proceedings of ACL 2013, pp ACL Language and Information Technologies, School of Computer Science, Carnegie Mellon University, Master of Science 2004 ATR NTT NICT e

24 Vol. 22 No. 4 December ACL ACM

2

NTT 2012 NTT Corporation. All rights reserved. 2 3 4 5 Noisy Channel f : (source), e : (target) ê = argmax e p(e f) = argmax e p(f e)p(e) 6 p( f e) (Brown+ 1990) f1 f2 f3 f4 f5 f6 f7 He is a high school