18301.dvi

Size: px

Start display at page:

Download "18301.dvi"

あかりたつざわ
5 years ago
Views:

1 Cognitive Studies, 18(3), (Sep. 2011) Information from face and voice plays an important role in social communication. As shown in the study of speech perception, facial and vocal signals are integrated even in the perception of emotion. This paper reviews the studies on multisensory perception of emotion by faces and voices. This paper then introduces recent studies on the cultural differences in the multisensory perception of emotion. It is emphasized that the combination of faces and voices can yield the richness in the expressions of emotions. Keywords: multisensory communication, multisensory perception, emotion perception, facial expression, vocal expression, cultural difference 1. ( Campbell, 2007;, 2011 ) 2 3 Multisensory Communication of Emotion by Faces and Voices, by Akihiro Tanaka (Waseda Institute for Advanced Study)

2 Vol. 18 No Darwin (Ekman, 1973) ( Sherer, 2003) ( de Gelder, 2009) (de Gelder, 2006) (2011) de Gelder and Vroomen (2000) Massaro (1996) de Gelder and Vroomen (2000) % 100% 1 2

3 418 Cognitive Studies Sep Vroomen, Driver, and de Gelder (2001) de Gelder, Pourtois, and Weiskrantz (2002) (Collignon, Girard, Gosselin, Roy, Saint-Amour, Lassonde, & Lepore, 2008;, 1993;, 2008;, 2004) (Koizumi, Tanaka, Imai, Hiramatsu, Hiramoto, & de Gelder, 2011) (de Gelder, Vroomen, de Jong, Masthoff, Trompenaars, & Hodiamont, 2005; de Jong, Hodiamont, van den Stock, & de Gelder, 2009) ms (de Gelder, Bocker, Tuomainen, Hensen, & Vroomen, 1999) 110 ms N1 (Pourtois, de Gelder, Vroomen, Rossion, & Crommelinck, 2000) 240 ms 20 ms (Pourtois, Debatisse, Despland, & de Gelder, 2002) Dolan, Morris, and de Gelder (2001) fmri (functional Magnetic Resonance Imaging: ) Ethofer, Anders, Erb, Droll, Royen, Saur, Reiterer, Grodd, and Wildgruber (2006) fmri Pourtois, de Gelder, Bol, and Crommelinck (2005) PET (Positron Emission Tomography: ) (supra-additive) Kreifelts, Ethofer, Grodd, Erb, and Wildgruber (2007) fmri 7

4 Vol. 18 No Ekman (Radeau & Bertelson, 1977; Thurlow & Jack, 1973) (Thurlow & Jack, 1973) (, 2010; Kamachi, Bruce, Mukaida, Gyoba, Yoshikawa, & Akamatsu, 2001; Sato & Yoshikawa, 2004) (Hirsh & Sherrick, 1961; Zampini, Shore, & Spence, 2003) (Tanaka, Sakamoto, Tsumura, & Suzuki, 2009; van Wassenhove, Grant, & Poeppel, 2007) 50 ms 200 ms (Ekman, 1972; Ekman & Friesen, 1971) (Elfenbein & Ambady, 2002; Jack, Blais, Scheepers, Schyns, & Caldara, 2009; Yuki, Maddux, & Masuda, 2007) (Ekman & Friesen, 1975) Masuda, Ellsworth, Mesquita, Leu, Tanida, and van de Veerdonk (2008)

5 420 Cognitive Studies Sep Yuki et al. (2007) Ishii, Reyes, and Kitayama (2003) ( 2004) 4.2 (Tanaka, Koizumi, Imai, Hiramatsu, Hiramoto, & de Gelder, 2010a) 1) )

6 Vol. 18 No Tanaka et al. (2010a) (Tanaka, Koizumi, Imai, Hiramatsu, Hiramoto, & de Gelder, 2010b) 100% 4.3 (Sekiyama & Tohkura, 1991, 1993) (Ishii et al., 2003) 2 Tanaka et al. (2010a)

7 422 Cognitive Studies Sep (Ekman, 1972; Matsumoto, Takeuchi, Andayani, Kouznetsova, & Krupp, 1998) (Markus & Kitayama, 1991) (2011) Ishii (2011) (2011) Van den Stock, Righart, and de Gelder (2007) (, ) 4

8 Vol. 18 No Ekman % 1 : McGurk & MacDonald, 1976) 6.

9 424 Cognitive Studies Sep. 2011, (Grossmann, Striano, & Friederici, 2006) (Kuraoka & Nakamura, 2007) (SCOPE) ( ) (2011).., 18, Campbell, R. (2007). The processing of audiovisual speech: Empirical and neural bases. Philosophical Transactions of the Royal Society B: Biological Sciences, 363, Collignon, O., Girard, S., Gosselin, F., Roy, S., Saint-Amour, D., Lassonde, M., & Lepore, F. (2008). Audio-visual integration of emotion expression. Brain Research, 1242, de Gelder, B. (2006). Towards the neurobiology of emotional body language. Nature Reviews Neuroscience, 7, de Gelder, B. (2009). Why bodies? Twelve reasons for including bodily expressions in affective neuroscience. Philosophical Transactions of the Royal Society B: Biological Sciences, 364, de Gelder, B., Bocker, K. B., Tuomainen, J., Hensen, M., & Vroomen, J. (1999). The combined perception of emotion from voice and face: Early interaction revealed by human electric brain responses. Neuroscience Letters, 260, de Gelder, B., Pourtois, G., & Weiskrantz, L. (2002). Fear recognition in the voice is modulated by unconsciously recognized facial expressions but not by unconsciously recognized affective pictures. Proceedings of the National Academy of Sciences USA, 99, de Gelder, B., & Vroomen, J. (2000). The perception of emotions by ear and by eye. Cognition & Emotion, 14,

10 Vol. 18 No de Gelder, B., Vroomen, J., de Jong, S. J., Masthoff, E. D., Trompenaars, F. J., & Hodiamont, P. (2005). Multisensory integration of emotional faces and voices in schizophrenics. Schizophrenia Research, 72, de Jong, J. J., Hodiamont, P. P. G., Van den Stock, J., & de Gelder, B. (2009). Audiovisual emotion recognition in schizophrenia: Reduced integration of facial and vocal affect. Schizophrenia Research, 107, Dolan, R. J., Morris, J. S., & de Gelder, B. (2001). Crossmodal binding of fear in voice and face. Proceedings of the National Academy of Sciences USA, 98, Ekman, P. (1972). Universals and cultural differences in facial expressions of emotion. In J. Cole (Ed.), Nebraska symposium on motivation, 1971, Lincoln: University of Nebraska Press. Ekman, P. (1973). Cross-cultural studies of facial expression. In P. Ekman (Ed.), Darwin and facial expression: A century of research in review. New York: Academic Press. Ekman, P., & Friesen, W. V. (1971). Constants across cultures in the face and emotion. Journal of Personality and Social Psychology, 17, Ekman, P., & Friesen, W. V. (1975). Unmasking the face. New Jersey: Prentice-Hall. Elfenbein, H. A., & Ambady, N. (2002). On the universality and cultural specificity of emotion recognition: A meta-analysis. Psychological Bulletin, 128, Ethofer, T., Anders, S., Erb, M., Droll, C., Royen, L., Saur, R., Reiterer, S., Grodd, W., & Wildgruber, D. (2006). Impact of voice on emotional judgment of faces: An eventrelated fmri study. Human Brain Mapping, 27, (2010).., 81 (4), Grossmann, T., Striano, T., & Friederici, A. D. (2006). Crossmodal integration of emotional information from face and voice in the infant brain. Developmental Science, 9, Hirsh, I. J. & Sherrick, J. E. (1961). Perceived Order in Different Sense Modalities. Journal of Experimental Psychology, 62, Ishii, K. (2011). Mere exposure to faces increases attention to vocal affect: Stroop interference in Japan and the US. Cognitive Studies, 18, Ishii, K., Reyes, J. A., & Kitayama, S. (2003). Spontaneous attention to word content versus emotional tone: Differences among three cultures. Psychological Science, 14, (1993).., 41 (1), Jack, R. E., Blais, C., Scheepers, C., Schyns, P. G., & Caldara, R. (2009). Cultural confusions show that facial expressions are not universal. Current Biology, 19, 1 6. Kamachi, M., Bruce, V., Mukaida, S., Gyoba, J., Yoshikawa, S., & Akamatsu, S. (2001). Dynamic properties influence the perception of facial expressions. Perception, 30, Koizumi, A., Tanaka, A., Imai, H., Hiramatsu, S., Hiramoto, E., Sato, T., & de Gelder, B. (2011). The effects of anxiety on the interpretation of emotion in the face-voice pairs. Experimental Brain Research, Online first, DOI: /s Kreifelts, B., Ethofer, T., Grodd, W., Erb, M. & Wildgruber, D. (2007). Audiovisual integration of emotional signals in voice and face: An event-related fmri study. NeuroImage, 37, Kuraoka, K., & Nakamura, K. (2007). Responses of single neurons in monkey amygdala to facial and vocal emotions. Journal of Neurophysiology, 97, Markus, H. R., & Kitayama, S. (1991). Culture and the self: Implications for cognition, emotion, and motivation. Psychological Review, 98, Massaro, D. W., & Egan, P. B. (1996). Perceiving affect from the voice and the face. Psychonomic Bulletin & Review, 3, Masuda, T., Ellsworth, P., Mesquita, B., Leu, J., Tanida, S., & van de Veerdonk, E. (2008). Placing the face in context: Cultural differences in the perception of facial emotion. Journal of Personality and Social Psychology, 94, Matsumoto, D., Takeuchi, S., Andayani, S., Kouznetsova, N., & Krupp, D. (1998). The contribution of individualism-collectivism to cross-national differences in display rules.

11 426 Cognitive Studies Sep Asian Journal of Social Psychology, 1, McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264, (2008).., 16 (3), (2011).., 18, Pourtois, G., Debatisse, D., Despland, P. A., & de Gelder, B. (2002). Facial expressions modulate the time course of long latency auditory brain potentials. Brain Research. Cognitive Brain Research, 14, Pourtois, G., de Gelder, B., Bol, A., & Crommelinck, M. (2005). Perception of facial expressions and voices and of their combination in the human brain. Cortex, 41, Pourtois, G., de Gelder, B., Vroomen, J., Rossion, B., & Crommelinck, M. (2000). The time-course of intermodal binding between seeing and hearing affective information. Neuroreport, 11, Radeau, M., & Bertelson, P. (1977). Adaptation to auditory-visual discordance and ventriloquism in semirealistic situations. Perception & Psychophysics, 22, Sato, W., & Yoshikawa, S. (2004). The dynamic aspects of emotional facial expressions. Cognition and Emotion, 18, Scherer, K. R. (2003). Vocal communication of emotion: A review of research paradigms. Speech Communication, 40, (2011). /., 18, Sekiyama, K., & Tohkura, Y. (1991). McGurk effect in non-english listeners: Few visual effects for Japanese subjects hearing Japanese syllables of high auditory intelligibility. Journal of the Acoustical Society of America, 90, Sekiyama, K., & Tohkura, Y. (1993). Interlanguage differences in the influence of visual cues in speech perception. Journal of Phonetics, 21, (2004).., 4, 髙 ( ) Tanaka, A., Koizumi, A., Imai, H., Hiramatsu, S., Hiramoto, E., & de Gelder, B. (2010a). I feel your voice: Cultural differences in the multisensory perception of emotion. Psychological Science, 21, Tanaka, A., Koizumi, A., Imai, H., Hiramatsu, S., Hiramoto, E., & de Gelder, B. (2010b). Cross-cultural differences in the multisensory perception of emotion. Proceedings of the International Conference on Auditory-Visual Speech Processing 2010, Tanaka, A., Sakamoto, S., Tsumura, K., & Suzuki, Y. (2009). Visual speech improves the intelligibility of time-expanded auditory speech. Neuroreport, 20, Thurlow, W. R., & Jack, C. E. (1973). Certain determinants of the ventriloquism effect. Perceptual and Motor Skills, 36, (2004).., 11 (2), Van den Stock, J., Righart, R., & de Gelder, B. (2007). Body expressions influence recognition of emotions in the face and voice. Emotion, 7, van Wassenhove, V., Grant, K. W., & Poeppel, D. (2007). Temporal window of integration in bimodal speech. Neuropsychologia, 45, Vroomen, J., Driver, J., & de Gelder, B. (2001). Is cross-modal integration of emotional expressions independent of attentional resources? Cognitive, Affective and Behavioral Neuroscience, 1 (4), Yuki, M., Maddux, W. W., & Masuda, T. (2007). Are the windows to the soul the same in the East and West? Cultural differences in using the eyes and mouth as cues to recognize emotions in Japan and the United States. Journal of Experimental Social Psychology, 43, Zampini, M., Shore, D. I., & Spence, C. (2003). Audiovisual temporal order judgments. Experimental Brain Research, 152, (Received 30 June 2011)

12 Vol. 18 No ( )

18301.dvi

18301.dvi Cognitive Studies, 18(3), 441-452. (Sep. 2011) 2 1 1, 2 3 The ability to recognize emotional states of others is a fundamental social skill. In this study, we investigated the extent to which complex emotions