203 図 2,re re, [Nivre 08]. y, 1 y i. ŷ = arg max y Y * J j=1 P r(y j y j 1 1,x) 2,, Pr y j y 1 j 1, x.,. ŷ = arg max y Y * 図 1 J j=1 exp(w o φ(y j,y j

Size: px

Start display at page:

Download "203 図 2,re re, [Nivre 08]. y, 1 y i. ŷ = arg max y Y * J j=1 P r(y j y j 1 1,x) 2,, Pr y j y 1 j 1, x.,. ŷ = arg max y Y * 図 1 J j=1 exp(w o φ(y j,y j"

かずまさたなせ
5 years ago
Views:

1 ニューラルネットワーク研究のフロンティアニューラルネットワークによる構造学習の発展 Advances in Structured Learning by Neural Networks 渡辺太郎 Taro Watanabe Google Inc. tarow@google.com Keywords: natural language processing, machine translation, parsing, neural network, deep learning, structured learning. 1. はじめに,.,, [Chen 14, Weiss 15].,,BBN [Devlin 14].,,,,,,,.1990,.,, [Bengio 13].,,,,,..,, 2.,3,,.,,. 4.,,,, 自然言語処理における統計モデル,I x x I 1 x 1 x 2 x I,x i X, y.,,y,,., x,y ŷ. ŷ = arg maxp r(y x) 1 y,y J y y J 1 y 1 y 2 y J, y j Y. 1 a.,7, I saw, two, black dogs, at the shrine 4 [Koehn 03]., 1 b, NN NP NN P, 12 [Klein 04].CYK,. 1 c.,,shift.,,

2 203 図 2,re re, [Nivre 08]. y, 1 y i. ŷ = arg max y Y * J j=1 P r(y j y j 1 1,x) 2,, Pr y j y 1 j 1, x.,. ŷ = arg max y Y * 図 1 J j=1 exp(w o φ(y j,y j 1 1,x)) y Y exp(wo φ(y,y j 1 1,x)) 3 φ q Y Y j 1 X * R q, w o R q. w o.,y argmax, φ w o.,, [Och 03]., [Petrov 06]., [Nivre 08].,,,.,. 3. ニューラルネットワークの応用.,φ,.,. p(y j y j 1 1,x) p(y j y j 1 j 2 ) 4 = exp(u yj (W o h j + b o )) y Y exp(uy (W o h j + b o )) 5 h j = f(w h z j + b h ) 6 z j = [W i u yj 2 ;W i u yj 1 ] 7 W i R q Y, y q embedding, u y {0, 1} Y y 1, ; [a ; b].z j R 2q W h R q 2q b h R q q, tanh sigmoid f.h j R q,w o R Y q, b o R Y,softmax. 2,y, 2 n-gram

3 [Bengio 03].,, Feed-Forward Neural Network FFNN. W o,z j. W o ; b o ; W h ; b h ; W i [Rumelhart 88] D 1. 8, 3 φ, w o. 5 W o 6,., 3-gram,O Y 3,O Y., 3,,.,,., [Liu 13]. n-gram [Schwenk 07], Noise Contrastive Estimate NCE [Gutmann 12] [Vaswani 13]., 5 softmax [Andreas 15],, [Devlin 14]. convolutional neural network, [Meng 15], tensor [Setiawan 15],. FFNN [Yang 13], FFNN [Chen 14, Weiss 15]. 4. 動的な構造に基づくニューラルネットワーク,,.,,,,FFNN,.,,.,. FFNN.,FFNN,, null を [Vaswani 13].,,. 4 1 回帰型ネットワーク recurrent neural network, j. j, x 1 y j,. p(y j x j 1 ) = g(uyj (W o h j + b o )) 9 h j = f(w h [h j 1 ;W i u xj ] + b h ) 10,h j R q, h j 1 W i u x j Rq.,g softmax. j, x j 1 h j 1. Elman network [Elman 90], 3.[Mikolov 10],x j y j 1,., 1 δ a, b a b 1. 図 3

4 205 x j,,, [Auli 13, Kalchbrenner 13].,[Sundermeyer 14],,.[Wu 14], y j,.[tamura 14],,NCE, FFNN [Yang 13]. FFNN,, h j,.[auli 14],,.,,. h j,,h 1,.,,,. Long Short-Term Memory LSTM [Hochreiter 97],,,., Gated Recurrent Unit GRU [Cho 14] [Chung 14]. 4 2 再帰型ネットワーク,, 図 4 図 5. recursive neural networks,, Directed Acyclic Graph DAG. 4,, h p R q, h l h r. p(y p x rp l p ) = g(u yp (W o h p + b o )) 11 h p = f(w h [h l ;h r ] + b h ) 12,h p, r x p lp.,,,.[socher 12],,,., LSTM, LSTM,, [Tai 15].[Socher 13], W h,pcfg.[stenetorp 13],. 4 3 スタック型ニューラルネットワーク,, [Dyer 15, Watanabe 15]., j y j,h j 1 h j push 10, 9.,push, top,pop

5 , j,top h top h j push,top h j. [Das 92]. push pop [Grefenstette 15] LSTM.[Dyer 15]., swap [Ballesteros 15].[Watanabe 15],[Dyer 15].[Le 14],. 5. エンコーダデコーダモデル,., FFNN,.,, [Bahdanau 15, Kalchbrenner 13, Sutskever 14].,,. 6. x LSTM. h e i = f(w e [h e i+1;w ie u xi ] + b e ) 13,, x I x 1, s., h 0 e h 1 d,, /s,. ŷ j = arg maxp(y ŷ j 1 1,x) 14 y Y p(y y j 1 1,x) = g(u y (W o h d j + b o )) 15 h d j = f(w d [h d j 1;W id u yj 1 ] + b d ) 16., [Miikkulainen 90, Vinyals 15], [Berg 92].[Mayberry 99]. 5 1 注意モデル図 7,,.[Bahdanau 15] attention model,.,. h i = f( W e [ h i 1 ;W ie u xi ] + b e ) 17 h i = f( W e [ h i+1 ;W ie u xi ] + b e ) 18 図 6,.

6 207 c j = I i=1 α i,j [ h i ; h i ] 19 h j d,y j 1 h j d 1 c j, α i, j, h i; h i h d j 1.α i, j,j i,. 5 2 大規模語彙化,,,., 14, Y,.,,g softmax,y,.,, UNK. [Luong 15],UNK, UNK.,,,. [Jean 15],.,,.,,,,.,. 6. 今後の展望,.,,.,,[Auli 14],.,,,,.,,.,,.,,,,.,,,,.,,,,,..[Tamura 14] NCE,.[Socher 11] recursive autoencoder [Pollack 90],. [Li 13, Li 14] [Liu 14, Su 15, Zhang 14],.,..,, [Collins 04].,,,,

7 [Weiss 15].[Watanabe 15] k-best,.,,,,. 謝辞,.. 参考文献 [Andreas 15] Andreas, J. and Klein, D.: When and why are loglinear models self-normalizing?, NAACL-HLT2015, pp , Denver, Colorado 2015 [Auli 13] Auli, M., Galley, M., Quirk, C. and Zweig, G.: Joint language and translation modeling with recurrent neural networks, EMNLP 2013, pp , Seattle, Washington, USA 2013 [Auli 14] Auli, M. and Gao, J.: Decoder Integration and Expected BLEU training for recurrent neural network language models, ACL 2014, pp , Baltimore, Maryland 2014 [Bahdanau 15] Bahdanau, D., Cho, K. and Bengio,Y.: Neural machine translation by jointly learning to align and translate, ICLR [Ballesteros 15] Ballesteros, M., Dyer, C. and Smith, N. A.: Improved transition-based parsing by modeling characters instead of words with LSTMs, EMNLP 2015, pp , Lisbon, Portugal 2015 [Bengio 03] Bengio, Y., Ducharme, R.,Vincent, P. and Janvin, C.: A neural probabilistic language model, J. Machine Learning Research, Vol. 3, pp [Bengio 13] Bengio,Y., Courville, A. and Vincent, P.: Representation learning: A review and new perspectives, IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol. 35, No. 8, pp [Berg 92] Berg, G.: A connectionist parser with recursive sentence structure and lexical disambiguation, AAAI 92, pp [Chen14] Chen, D. and Manning, C.: A fast and accurate dependency parser using neural networks, EMNLP 2014, pp , Doha, Qatar 2014 [Cho 14] Cho, K., Merrienboer, van B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H. and Bengio, Y.: Learning phrase representations using RNN encoder, decoder for statistical machine translation, EMNLP 2014, pp , Doha, Qatar 2014 [Chung 14] Chung, J., Gülçehre, Ç., Cho, K. and Bengio, Y.: Empirical evaluation of gated recurrent neural networks on sequence modeling, CoRR, Vol. abs/ [Collins04] Collins, M. and Roark, B.: Incremental parsing with the perceptron algorithm, ACL 2004, pp , Barcelona, Spain 2004 [Das 92] Das, S., Giles, C. L. and Sun, Zheng, G.: Learning context-free grammars: Capabilities and limitations of a recurrent neural network with an external stack memory, Conf. of the Cognitive Science Society, pp [Devlin14] Devlin, J., Zbib, R., Huang, Z., Lamar, T., Schwartz, R. and Makhoul, J.: Fast and robust neural network joint models for statistical machine translation, ACL 2014, pp , Baltimore, Maryland 2014 [Dyer 15] Dyer, C., Ballesteros, M., Ling, W., Matthews, A. and Smith, N. A.: Transition-based dependency parsing with stacklong short-term memory, ACL 2015, pp , Beijing, China 2015 [Elman 90] Elman, J. L.: Finding structure in time, Cognitive Science, Vol. 14, No. 2, pp [Grefenstette 15] Grefenstette, E., Hermann, K. M., Suleyman, M. and Blunsom, P.: Learning to transduce with unbounded memory, CoRR, Vol. abs/ [Gutmann12] Gutmann, M. U. and Hyvärinen, A.: Noisecontrastive estimation of unnormalized statistical models, with applications to natural image statistics, J. Machine Learning Research, Vol. 13, No. 1, pp [Hochreiter 97] Hochreiter, S. and Schmidhuber, J.: Long shortterm memory, Neural Computation, Vol. 9, No. 8, pp [Jean15] Jean, S., Cho, K., Memisevic, R. and Bengio, Y.: On using very large target vocabulary for neural machine translation, ACL 2015, pp. 1-10, Beijing, China 2015 [Kalchbrenner 13] Kalchbrenner, N. and Blunsom, P.: Recurrent continuous translation models, EMNLP 2013, pp , Seattle, Washington, USA 2013 [Klein 04] Klein, D. and Manning, C. D.: Parsing and Hypergraphs, Bunt, H., Carroll, J. and Satta, G., eds., New Developments in Parsing Technology, pp , Kluwer Academic Publishers, Norwell, MA, USA 2004 [Koehn03] Koehn, P., Och, F. J. and Marcu, D.: Statistical phrasebased translation, NAACL 03, pp , Stroudsburg, PA, USA 2003 [Le 14] Le, P. and Zuidema, W.: The inside-outside recursive neural network model for dependency parsing, EMNLP 2014, pp , Doha, Qatar 2014 [Li 13] Li, P., Liu, Y. and Sun, M.: Recursive autoencoders for ITG-based translation, EMNLP 2013, pp , Seattle, Washington, USA 2013 [Li 14] Li, P., Liu, Y., Sun, M., Izuha, T. and Zhang, D.: A neural reordering model for phrase-based translation, COLING 2014, pp , Dublin, Ireland 2014 [Liu 13] Liu, L., Watanabe, T., Sumita, E. and Zhao, T.: Additive neural networks for statistical machine translation, ACL 2013, pp , Sofia, Bulgaria 2013 [Liu 14] Liu, S., Yang, N., Li, M. and Zhou, M.: A recursive recurrent neural network for statistical machine translation, ACL 2014, pp , Baltimore, Maryland 2014 [Luong 15] Luong, T., Sutskever, I., Le, Q., Vinyals, O. and Zaremba,W.: Addressing the rare word problemin neural machine translation, ACL 2015, pp , Beijing, China 2015 [Mayberry 99] Mayberry, M. R. and Miikkulainen, R.: SARDSRN: A neural network shift-reduce parser, IJCAI 99, pp , San Francisco, CA, USA 1999 [Meng 15] Meng, F., Lu, Z., Wang, M., Li, H., Jiang, W. and Liu, Q.: Encoding source language with convolutional neural network for machine translation, ACL 2015, pp , Beijing, China 2015 [Miikkulainen 90] Miikkulainen, R.: A PDP architecture for processing sentences with relative clauses, COLING 90, pp , Stroudsburg, PA, USA 1990 [Mikolov 10] Mikolov, T., Karafit, M., Burget, L., Cernock, J. and Khudanpur, S.: Recurrent neural network based language model, INTERSPEECH 2010, pp [Nivre 08] Nivre, J.: Algorithms for deterministic incremental dependency parsing, Computational Linguistics, Vol. 34, No. 4, pp [Och03] Och, F. J.: Minimum error rate training in statistical machine translation, ACL 2003, pp , Sapporo, Japan 2003 [Petrov 06] Petrov, S., Barrett, L., Thibaux, R. and Klein, D.: Learning accurate, compact, and interpretable tree annotation, ACL 2006, pp , Sydney, Australia 2006 [Pollack 90] Pollack, J. B.: Recursive distributed representations,

209 Artificial Intelligence, Vol. 46, No. 1-2, pp. 77-105 1990 [Rumelhart 88] Rumelhart, D. E., Hinton, G. E. and Williams, R. J.

8 209 Artificial Intelligence, Vol. 46, No. 1-2, pp [Rumelhart 88] Rumelhart, D. E., Hinton, G. E. and Williams, R. J.: Neurocomputing: Foundations of Research, chapter Learning Representations by Back-propagating Errors, pp , MIT Press, Cambridge, MA, USA 1988 [Schwenk 07] Schwenk, H.: Continuous space language models, Computer Speech and Language,Vol. 21, No. 3, pp [Setiawan 15] Setiawan, H., Huang, Z., Devlin, J., Lamar, T., Zbib, R., Schwartz, R. and Makhoul, J.: Statistical machine translation features with multitask tensor networks, ACL 2015, pp , Beijing, China 2015 [Socher 11] Socher, R., Pennington, J., Huang, E. H., Ng, A. Y. and Manning, C. D.: Semi-supervised recursive autoencoders for predicting sentiment distributions, EMNLP 2011, pp , Edinburgh, Scotland, UK 2011 [Socher 12] Socher, R., Huval, B., Manning, C. D. and Ng, A. Y.: Semantic compositionality through recursive matrix-vector spaces, EMNLP 2012, pp , Jeju Island,Korea 2012 [Socher 13] Socher, R., Bauer, J., Manning, C. D. and Andrew, Y., N.: Parsing with compositional vector grammars, ACL 2013, pp , Sofia, Bulgaria 2013 [Stenetorp 13] Stenetorp, P.: Transition-based dependency parsing using recursive neural networks, Deep Learning Workshop at NIPS 2013, Lake Tahoe, Nevada, USA 2013 [Su 15] Su, J., Xiong, D., Zhang, B., Liu,Y., Yao, J. and Zhang, M.: Bilingual correspondence recursive autoencoder for statistical machine translation, EMNLP 2015, pp , Lisbon, Portugal 2015 [Sundermeyer 14] Sundermeyer, M., Alkhouli, T., Wuebker, J. and Ney, H.: Translation modeling with bidirectional recurrent neural networks, EMNLP 2014, pp , Doha, Qatar 2014 [Sutskever 14] Sutskever, I., Vinyals, O. and Le, Q. V.: Sequence to sequence learning with neural networks, Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N. D. and Weinberger, K. Q., eds., NIPS 2014, pp [Tai 15] Tai, K. S., Socher,R. and Manning, C. D.: Improved semantic representations from tree-structured long short-term memory networks, ACL 2015, pp , Beijing, China 2015 [Tamura 14] Tamura, A., Watanabe, T. and Sumita, E.: Recurrent neural networks for word alignment model, ACL 2014, pp , Baltimore, Maryland 2014 [Vaswani13] Vaswani, A., Zhao, Y., Fossum, V. and Chiang, D.: Decoding with large-scale neural language models improves translation, EMNLP 2013, pp , Seattle,Washington, USA 2013 [Vinyals 15] Vinyals, O., Kaiser, L., Koo, T., Petrov, S., Sutskever, I. and Hinton, G.: Grammar as a foreign language, Cortes, C., Lawrence, N., Lee, D., Sugiyama, M. and Garnett, R., eds., NIPS 2015, pp , Curran Associates, Inc [Watanabe 15] Watanabe,T. and Sumita, E.: Transition-based neural constituent parsing, ACL 2015, pp , Beijing, China 2015 [Weiss 15] Weiss, D., Alberti, C., Collins, M. and Petrov, S.: Structured training for neural network transition-based parsing, ACL 2015, pp , Beijing, China 2015 [Wu 14] Wu, Y., Watanabe, T. and Hori, C.: Recurrent neural network-based tuple sequence model for machine translation, COLING 2014, pp , Dublin, Ireland 2014 [Yang 13] Yang, N., Liu, S., Li, M., Zhou, M. and Yu, N.: Word alignment modeling with context dependent deep neural network, ACL 2013, pp , Sofia, Bulgaria 2013 [Zhang 14] Zhang, J., Liu, S., Li, M., Zhou, M. and Zong, C.: Bilingually-constrained phrase embeddings for machine translation, ACL 2014, pp , Baltimore, Maryland 著者紹介渡辺太郎 Language and Information Technologies, School of Computer Science,Carnegie Mellon University,Master of Science ATR NTT,NICT,,.,.

Computational Semantics 1 category specificity Warrington (1975); Warrington & Shallice (1979, 1984) 2 basic level superiority 3 super-ordinate catego

Computational Semantics 1 category specificity Warrington (1975); Warrington & Shallice (1979, 1984) 2 basic level superiority 3 super-ordinate category preservation 1 / 13 analogy by vector space Figure