日本感性工学会論文誌

Size: px

Start display at page:

Download "日本感性工学会論文誌"

とよみひめい
4 years ago
Views:

1 J-STAGE Transactions of Japan Society of Kansei Engineering J-STAGE Advance Published Date: doi: /jjske.TJSKE-D Automatic Affective Image Captioning System using Emotion Estimation Yuya MIYOSHI* and Masafumi HAGIWARA** * Hitachi, Ltd., Nakazato, Odawara-shi, Kanagawa , Japan ** Keio University, Hiyoshi, Kohoku-ku, Yokohama, Kanagawa , Japan Abstract : Image captioning has been actively studied these days, however, most of the systems output captions of factual expression. In this paper, an automatic affective image captioning system using emotion estimation is proposed. The proposed system consists of four parts: a base caption generation part composed by the conventional CNN (VGG16), a scene estimation part, an emotion estimation part, and a figurative expression generation part. When a human exists in an image, the emotion is estimated from his/her facial expression and simile is used. When a human does not exist in an image, personification of metaphor is used. Evaluation experiments have been carried out using three kinds of evaluation indexes; BLUE, METEOR, and CIDEr. The experimental results indicate the effectiveness of the proposed system to generate affective captions. Keywords : Image captioning, Convolutional neural network, Emotion estimation 1. Image Caption 2 Image Caption Image Caption 1-5 Convolutional Neural Network CNN 6 Recurrent Neural Network RNN 7 8 Image Caption Image Caption Image Caption Yu 9 Yu CNN CNN Image Caption Image Caption Microsoft COCO 10 Fricker8k 11 Fricker30k 12 Image Caption Mathews 13 S V O C S V C V Received: / Accepted: Copyright 2019 All Rights Reserved.

2 Fu 5 CNN RNN 1S t t N S 0 S N W I W e p t t CNN VGG16 18 VGG16 ILSVRC CNNRNN Long short-term memory LSTM I CNN F Y 1 2. LSTM 2 x t 0, t, N 1 LSTM S t W e LSTM Y W I X t S t t 0 N 1 p t

3 4. 5 T n n Microsoft COCO 10 Image Caption Latent Dirichlet Allocation LDA 21 Multilayer Perceptron MLP 2.2 LDA MLP LDA 80 LDA 80 2 MLP 2 Image Net 22 CNN CNN ILSVRC AlexNet 23 Place MS COCO Cake A woman Vase Stand Sitting Suitcase Grass Snow Road sign Water Fly Jump Various Suit, tie Bear umbrella Train Car Black and white Pizza Banana Two Room Phone Three Traffic signal Skateboard Platform Close up Bike Cute Eat On A Computer Swing Kite Ride Bird Sandwich Park Elephant Line Dog Walk Window In Fruits Frisbee Bed Controller Tennis Zebra Child Prepare Head A man Bus Vegetable Hydrant Plane Clock Cut Mirror Bench Hold Group Old Surfboard Two people Tree Bathroom Baseball Donut Kitchen Toilet Horse Team Picture cat 2 Place MLP % 1 MS COCO LDA Microsoft Emotion API 5 Emotion API Happiness Sadness Angry Disgust Contempt Fear Surprise Neutral CNN Happiness Sadness Disgust Fear 5,000 30,

4 a 1 2 Emotion API 8 1 word2vec cos cos stand Happiness happily really honestly Sadness sadly frankly painfully Angry angrily rightfully violently Contempt contemptibly justly sternly Disgust calmly irritably wearily Fear fearfully eventually easily surprise surprisingly unexpectedly exactly Neutral sharply straight carefully 3 0% 25% 90% 25% 50% 50% 75% 75% % cheerfully joyfully happily Happiness brightly gladly happily Sadness sadly sadly Angry nervously wildly violently fiercely fiercely nervously angrily Contempt contemptibly sternly contemptibly Disgust awkwardly wearily irritably disgustingly awkwardly wearily irritably disgustingly Fear anxiously deliberately fearfully anxiously fearfully Surprise surprisingly surprisingly Neutral normally seriously calmly ordinary normally run 2 2 Happiness Sadness Angry Disgust Contempt Fear Surprise Neutral 8 0%~25% 25% 50% 50% 75% 75% 90% 90% 100% 5 2 cos cos 3 b Happiness Neutral 80 9 as if ~ Happiness Sadness Fear Disgust BLEU 15 METEOR 16 CIDEr 17

4 82,783 40,504 1,000 3 4 BLEU n-gram METEOR BLEU

CIDEr Microsoft COCO 1 CIDEr TF-IDF 28 3 Mathews

3.2.1 3 20 11 3 14 1 10 5 1 5 1 5 1 5 1 5

CIDEr Mathews 14 50.0 31.2 20.3 13.1 16.8 61.8 70.

5 4 82,783 40,504 1, BLEU n-gram METEOR BLEU BLEU METEOR F BLEU METEOR WordNet 27 BLEU METEOR CIDEr Microsoft COCO 1 CIDEr TF-IDF 28 3 Mathews 13 Microsoft COCO 4 5 Mathews Microsoft COCO 5 BLEU1 BLEU2 BLEU3 BLEU4 METEOR CIDEr Mathews

6 p < p < LDA 20 MLP fearfully sadly 4. CNN LSTM 2

7 2 1 Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., and Darrell, T.: Long-term recurrent convolutional networks for visual recognition and description, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp , Vinyals, O., Toshev, A., Bengio, S., and Erhan, D.: Show and tell: a neural image caption generator, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp , Fang, H., Gupta, S., Iandola, F., Srivastava, R. K., Deng, L., Dollar, P., Gao, J., He, X., Mitchell, M., Platt, J. C., Zitnick, C. L., and Zweig, G.: From captions to visual concepts and back, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp , Tran, K., He, X., Zhang, L., Sun, J., Carapcea, C., Thrasher, C., Buehler, C., and Sienkiewicz, C.: Rich image captioning in the wild, IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp , Fu, K., Jin, J., Cui, R., Sha, F., and Zhang, C.: Aligning where to see and what to tell: image captioning with region-based attention and scene-specific contexts, IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(12), pp , Lecun, Y., Bottou, L., Bengio, Y., and Haffner, P.: Gradientbased learning applied to document recognition, Proceedings of the IEEE, 86(11), pp , Cho, K., van Merrienboer, B., Gulcehre, C., Bougares, F., Schwenk, H., and Bengio, Y.: Learning phrase representations using RNN encoder-decoder for statistical machine translation, Conference on Empirical Methods in Natural Language Processing (EMNLP), pp , Sutskever, I., Vinyals, O., and Le, Q. V.: Sequence to sequence learning with neural networks, Advances in Neural Information Processing Systems, 27, pp , Yu, Z., and Zhang, C.: Image based static facial expression recognition with multiple deep network learning, In Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, pp , Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollar, P., and Zitnick, C. L.: Microsoft COCO: common objects in context, European Conference on Computer Vision (ECCV) 2014, pp , Rashtchian, C., Young, P., Hodosh, M., and Hockenmaier, J.: Collecting image annotations using Amazon s Mechanical Turk, Proc. NAACL HLT Workshop Creating Speech Language Data Amazon s Mech. Turk, pp , Young, P., Lai, A., Hodosh, M., and Hockenmaier, J.: From image descriptions to visual denotations: new similarity metrics for semantic inference over event descriptions, Transactions of the Association for Computational Linguistics, 2, pp.67-78, Mathews, A., Xie, L., and He, X.: SentiCap: generating image descriptions with sentiments, AAAI, pp Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J.: BLEU: a method for automatic evaluation of machine translation, In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL 02, pp , Denkowski, M. J. and Lavie, A.: Meteor universal: language specific translation evaluation for any target language, Proceedings of the Ninth Workshop on Statistical Machine Translation, pp , Vedantam, R., Zitnick, C. L., and Parikh, D.: CIDEr: consensus based image description evaluation, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp , Simonyan, K., and Zisserman, A.: Very deep convolutional networks for large-scale image recognition, CoRR abs/ , Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L.: ImageNet large scale visual recognition challenge, International Journal of Computer Vision, 115(3), pp , Hochreiter, S., and Schmidhuber, J.: Long short-term memory, Neural Computation, 9(8), pp , Blei, D. M., Ng, A. Y., Jordan, M. I.: Latent dirichlet allocation, Journal of Machine Learning Research, 3, pp , Deng, J., Dong, W., Socher, R., Li, L. J., Li, K., and Fei-Fei, L.: ImageNet: a large-scale hierarchical image database, IEEE

Conference on Computer Vision and Pattern Recognition, pp.248-255, 2009. 23 Krizhenvshky, A., Sutskever, I., and Hinton, G.

: Learning deep features for scene recognition using places database, Advances in Neural Information Processing Systems, 27, pp.487-495, 2014. 25 Mikolov, T., Yih, S. W.-T., and Zweig, G.

8 Conference on Computer Vision and Pattern Recognition, pp , Krizhenvshky, A., Sutskever, I., and Hinton, G.: ImageNet classification with deep convolutional networks, Advances in Neural Information Processing Systems, 25, pp , Zhou, B., Lapedriza, A., Xiao, J., Torralba, A., and Oliva, A.: Learning deep features for scene recognition using places database, Advances in Neural Information Processing Systems, 27, pp , Mikolov, T., Yih, S. W.-T., and Zweig, G.: Linguistic regularities in continuous space word representations, In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT-2013), pp , Mikolov, T., Chen, K., Corrado, G., and Dean, J.: Efficient estimation of word representations in vector space, arxiv: [cs.cl], Miller, G. A., Beckwith, R., Fellbaum, C., Gross, D., and Miller, K.: WordNet: an on-line lexical database, International Journal of Lexicography, 3, pp , Robertson, S.: Understanding inverse document frequency: on theoretical arguments for IDF, Journal of Documentation, 60(5), pp , IEEE Consumer Electronics Society

Computational Semantics 1 category specificity Warrington (1975); Warrington & Shallice (1979, 1984) 2 basic level superiority 3 super-ordinate catego

Computational Semantics 1 category specificity Warrington (1975); Warrington & Shallice (1979, 1984) 2 basic level superiority 3 super-ordinate category preservation 1 / 13 analogy by vector space Figure