26 fmri Decoding of Human Emotion induced by Visual Stimuli using fmri Brain Signal 1150320 2015 2 27
fmri BCI(Brain Computer Interface) IAPS fmri(fnuctional Magnetic Resonace Imaging) Valence( ) Arousal( ) IAPS Valence ( ) 2 BCI 48 fmri SD(Semantic Differential) SPM(Statistical Parametric Mapping) p 0.001 t 2 SVM(Support Vector Machine) 5 80 i
BCI,, fmri(fuctional Magnetic Resonace Imaging), SVM(Support Vector Machine) ii
Abstract Decoding of Human Emotion induced by Visual Stimuli using fmri Brain Signal Hirokazu Takahashi Estimation of human emotion from brain activities has been studied. These studies aim at applications to BCI(Brain Computer Interface). A previous study shows that the relation between presented image of IAPS(International Affective Picture System) to human subjects and their emotion. The study uses fmri(functional Magnetic Resonance Imaging) to measure brain activities. It uses the regression analysis to estimate the emotion from the measurement data output from the fmri and indicators of valence and arousal of IAPS. The result shows that low scored value data of the brain activation level have no correlation to the emotion, however, high scored value data of the brain activation level have the possibility of correlation to the emotion. In this thesis, these high scored measurement value data are focused. In the experiment, human subjects are presented the IAPS image whose indicators of valence is high. The output brain activation data is expected as high. This study verifies comfort-discomfort images can be identifiable. Previous researches of human emotion show that some brain area plays an important role in human emotion. However, in the application to BCI, we do not need to care of the brain area of emotion because the only accurate classification is required. Therefore, in this research, the all brain area is used to perform machine learning. In the experiment, 48 comfort-discomfort images are presented at random to human subjects, we measure the subjects brain activity by fmri. After the experiment, iii
we do a questionnarie to the subjects using SD(Semantic Differential) method to select the subjective comfort-discomfort images of each subject to be used for analysis. The voxel values and their location in the brain are analyzed using t-test(significant level p < 0.001) to find the significant difference at the activation. The voxel values are learned by a machine learning, and are evaluated by cross-validation, and finally the accuracy of the recognition is calculated. The result shows that the subjective accuracy of comfort-discomfort images of 5 subjects has 80 in average. From the result, the comfort-discomfort feelings that induced from the visual stimuli can be estimated from the brain information. The result of this paper shows the possibility of the BCI which translates the human emotion. key words Brain infromation decording, Comfort, Discomfort, Human emotion, functional-magnetic-resonance-imaging, Support-Vector-Machine iv
1 1 2 3 2.1................................ 3 2.2............................ 5 3 6 3.1 fmri..................... 6 3.1.1................................ 6 3.1.2................................. 6 3.1.3................................ 7 3.1.4................................ 7 3.1.5............................. 9 3.1.6............................ 10 3.2 SPM............................... 11 3.2.1................................ 11 3.2.2................................. 11 3.2.3................................ 11 3.2.4................................ 12 3.3 SVM...................... 12 4 14 4.1........................ 14 4.1.1 A................................ 17 4.1.2 B................................ 18 v
4.1.3 C................................ 19 4.1.4 D................................ 20 4.1.5 E................................ 21 4.2................................. 22 5 24 25 28 A : 30 B 32 C 40 vi
3.1............................. 8 3.2............................. 9 3.3 (1 )............................ 9 3.4................................ 10 4.1............................... 15 4.2 A.................... 17 4.3 B........................ 18 4.4 B...................... 18 4.5 C........................ 19 4.6 C...................... 19 4.7 D........................ 20 4.8 D...................... 20 4.9 E........................ 21 4.10 E...................... 21 4.11......................... 22 4.12....................... 22 4.13........................ 23 4.14....................... 23 A.1................... 31 C.1 A p1............... 41 C.2 A p2............... 42 C.3 B.................... 43 vii
C.4 B................... 44 C.5 C p1.................. 45 C.6 C p2.................. 46 C.7 C p3.................. 47 C.8 C p1................. 48 C.9 C p2................. 49 C.10 C p3................. 50 C.11 C p4................. 51 C.12 D.................... 52 C.13 D p1................. 53 C.14 D p2................. 54 C.15 D p3................. 55 C.16 E.................... 56 C.17 E p1................. 57 C.18 E p2................. 58 viii
4.1.............. 16 4.2............................. 16 B.1 IAPS.................. 33 B.2 IAPS.................. 34 B.3 A............ 35 B.4 B............ 36 B.5 C............ 37 B.6 D............ 38 B.7 E............ 39 ix
1 [1] [2] [3] [4] [1] BCI(Brain Computer Interface) fmri(functional Magnetic Resonance Imaging) [5] IAPS[6] fmri Valence( ) Arousal( ) IAPS Valence ( ) Valence ( ) Valence 2 SVM(Support 1
Vector Machine) 5 80 2 SVM 3 fmri SPM(Statistical Parametoric Mapping) SVM( ) 4 5 2
2 2.1 NIRS PET fmri NIRS (oxy-hb) NIRS [7] oxy-hb oxy-hb NIRS [8] 3
2.1 PET PET [9] fmri (MR ) fmri [10] BCI fmri NIRS PET 4
2.2 2.2 [11] SVM(Support Vector Machine) SVM 1995 AT&T V.Vapnik SVM 2 2 2 SVM 2 2 SVM 5
3 3.1 fmri fmri 3.1.1 IAPS[6] 24 24 48 fmri A 3.1.2 22 5 ( 3 2 ) fmri MRI 6
3.1 fmri 3.1.3 SIEMENS fmri MAGNETOM Verio3T 3T 1 45mT/m 1 200mT/m/ms (FoV) 50cm 3.1.4 Neurobehavioral Systems Presentation[13] (International Affective Picture System: IAPS)[6] IAPS 700 1 9 Valence Arousal Arousal Valence : Valence 7.0 Arousal 5.0 : Valence 3.0 Arousal 5.0 7
3.1 fmri Valence 1 9 7 9 1 3 Arousal 5 3.1 3.2 [14] B 3.1 8
3.1 fmri 3.2 3.1.5 3.3 15 (5 ) 6 15 (5 ) 9 (3 ) 1 8 BOLD [5] 1 165 4 12 3.3 (1 ) 9
3.1 fmri 3.1.6 IAPS [14] SD(Semantic Differential) 7 3.4 3.4 2 10
3.2 SPM 3.2 SPM SPM(Statistical Parametric Mapping)[15] SPM8 SPM5 3.2.1 fmri DICOM SPM Analyze MRIConvert[16] 3.2.2 Analyze SPM8 Realignment Normalisation Smoothing Realignment fmri Normaliseation Realignment Smoothing fmri 6 (2 ) 3.2.3 SPM5 2 (GLM:General Linear Model) GLM 2 (contrast) p 0.001 t 2 11
3.3 SVM 1. 2. 1 2 p 0.001 MNI Talairach Daemon[17] 3.2.4 5 SPM5 p 0.001 t MNI Talairach Daemon 3.3 SVM 2 ROI SVM( ) BDTB1.2.2(Brain Decorder Toolbox)[18] libsvm-3.16[19] SVM(Support Vector Machine) 12
3.3 SVM SVM( ) 1. 2 2. 2 13
4 4.1 SVM( ) 4.1 4.1 4.2 4.1 IAPS SD 65 80 B B ROI 4.1.2 C D 2 4.1 C 4 4.2 IAPS 4.1 C E 4.1.1 14
4.1 SD IAPS IAPS 4.1 15
4.1 4.1 ( ) ( ) ( ) A 71(10/14) 83(15/18) 78(25/32) B 50(6/12) 85(17/20) 71(23/32) C 80(8/10) 78(11/14) 79(19/24) D 70(7/10) 92(13/14) 83(20/24) E 76(10/13) 90(10/11) 83(20/24) 4.2 ( ) ( ) ( ) A 62(15/24) 62(15/24) 62(30/48) B 54(13/24) 50(12/24) 52(25/48) C 75(18/24) 78(18/24) 75(36/48) D 70(17/24) 75(18/24) 72(35/48) E 58(14/24) 54(13/24) 56(27/48) ROI ROI t 16
4.1 4.1.1 A A p 0.001 SPM 2 ROI 4.2 ROI 2 1. MNI (-12, 52, -8) 200 t 4.29 10 2. MNI (-20, -66, -50), 29 t 3.48 5mm 4.2 A 17
4.1 4.1.2 B B p 0.001 SPM 4.3 4.4 ROI 3 ROI MNI (-42, 22, 52) 22 t 3.50 8 ROI MNI (36, -58, 68) 48 t 3.80 5mm MNI (-4, -70, -30) 13 t 3.44 ( ) 4.3 B 4.4 B 18
4.1 4.1.3 C C p 0.001 SPM 4.5 4.6 ROI 2 ROI MNI (-18, -90, -44) 200 t 6.00 5mm ROI MNI (38, 32, -22) 96 t 5.03 47 4.5 C 4.6 C 19
4.1 4.1.4 D D p 0.001 SPM 4.7 4.8 ROI 3 ROI MNI (-18, -58, 20) 24 t 3.36 31 ROI MNI (-46-60 -14) 200 t 5.68 37 MNI (48 4 56) 200 t 5.42 6 4.7 D 4.8 D 20
4.1 4.1.5 E E p 0.001 SPM 4.9 4.10 ROI 4 ROI MNI (46-54 -40) 75 t 3.81 ( ) ROI MNI (-56, -12, 42) 200 t 4.91 3 MNI (52 18 14) 200 t 4.85 ( ) 45 MNI (52-54 -8) 200 t 4.62 37 4.9 E 4.10 E 21
4.2 4.2 5 2 p 0.001 SPM 2 4.11 4.14 4.11 MNI (14, -34, 20) [20] 4.12 MNI (30, -34, 14) [21] 4.11 4.12 22
4.2 4.12 MNI (24, -12, -6) ( ) - - 4.14 (24, 2, -18) 4.13 4.14 23
5 [5] SD IAPS 80 BCI 24
2 fmri SPM BDTB 2 4 1 ITNews 4 2 25
4 4 1 4 4 26
3 5 27
[1],,,,,,, Mayer -,, Vol.25 (2004) No.1 P41-49, 2004. [2],,, Vol.25 (2005) No.3 P233-241 2007. [3],,,, -, Vol.124 (2004) No.1 P213-214 2004. [4],, Vol.58 (2010) No.4 P464-479 2012. [5], fmri, 25, 2014 [6] Lang, Peter J., Margaret M. Bradley, Bruce N. Cuth-bert, International affective picture system (IAPS):Technical manual and affective ratings, 1999. [7],, 46, 2013. [8],,,,,, 2011. [9],, Vol.25 (2005) No.2 P139-144, 2006. [10], : fmri, 28(1), 17-27, 2010. 28
[11], fmri, 24, 2013 [12],, http://home.hiroshimau.ac.jp/tkurita/lecture/svm.pdf [13] Presentation, neurobehavioralsystems, http://www.neurobs.com/ [14],,, ( ), 2002. [15] Statistical Parametric Mapping, http://www.fil.ion.ucl.ac.uk/spm/ [16] MRIConvert files LCNI, http://lcni.uoregon.edu/downloads/mriconvert [17] Talairach Daemon,Research Imaging Institute of the University of Texas Health Science Center San Antonio, http://www.talairach.org/index.html [18] Brain Decoder Toolbox, ATR-DNi, http://www.cns.atr.jp/dni/download/braindecoder-toolbox/, 2014. [19] Chih-Chung Chang and Chih-Jen Lin, LIBSVM : a library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2:27:127:27,2011. Software available at http://www.csie.ntu.edu.tw/ cjlin/libsvm [20],,, 25 03,, 2003. [21],,,,,,, Vol.37 (2006) No.1 JANUARY P9-14, 2006. 29
A : A SPM p 0.001 2 SVM( ) A.1 D 80 50 30
A.1 31
B B IAPS 48 IAPS Valence Arousal B.1 B.2 B.3 B.7 32
B.1 IAPS Slide No. Valence Arousal Slide No. Valence Arousal 1710 8.02 5.53 2703 2.33 5.73 2340 7.65 5.35 2811 2.84 6.31 4002 7.69 7.15 3000 1.69 6.74 4007 7.7 7.39 3001 2.1 5.85 4008 7.75 6.94 3015 1.83 5.54 4071 7.37 6.78 3053 1.5 6.2 4141 7.46 6.73 3064 1.78 5.44 4225 7.57 6.94 3069 2.1 6.7 4664 7.99 7.72 3071 2.06 6.61 4680 7.73 5.94 3080 1.63 6.84 5260 7.47 6 3100 1.88 5.88 5621 7.28 6.96 3170 1.77 6.79 5623 7.12 5.56 3195 2.56 6.23 5700 7.7 5.94 3266 1.98 5.85 7230 7.42 5.81 6230 2.73 7.1 7350 7.08 5.4 6313 2.43 6.54 7405 7.08 6.03 6415 2.81 5.86 7502 7.3 5.74 6520 2.45 5.85 7508 7 5.1 7380 2.61 5.53 8030 7.29 7.32 9040 1.88 5.1 8185 7.32 7.06 9322 2.64 5.8 8190 8.13 6.41 9410 1.96 6.38 8380 7.25 6.02 9414 2.43 6.36 8420 7.61 5.71 9921 2.6 6.09 33
B.2 IAPS Slide No. Valence Arousal Slide No. Valence Arousal 1463 7.81 5.11 1052 2.99 6.89 1710 8.59 5.31 1201 2.93 6.87 2150 8.31 5.29 1271 2.65 6.01 2155 7.17 5.56 1274 2.86 5.66 2209 7.95 5.91 1525 2.67 6.86 2224 7.63 5.06 2095 1.48 5.72 2347 8.35 5.88 2800 1.41 5.87 4525 7.67 6.7 3000 1.21 7.77 4535 7.06 5.58 3103 1.71 6.6 4572 7.52 6.3 3150 1.98 6.94 4575 7.61 6.12 3170 1.2 7.55 4597 7.23 6.06 3213 2.61 6.79 4599 7.23 5.64 3400 2.06 7.12 4614 7.71 5.38 6311 2.36 5.12 4641 7.21 5.33 6350 1.44 7.52 5629 7.15 6.52 6360 1.86 6.82 5825 8.02 5.58 6560 1.78 6.86 5833 8.27 5.14 6571 2.15 5.87 7508 7.03 5.06 9181 1.98 6.09 7650 7.03 6.21 9253 1.6 5.65 8041 7.48 5.97 9295 2.31 5.28 8163 7.38 6.53 9326 1.84 6.28 8190 8.08 6.16 9410 1.2 7.54 8499 7.7 5.56 9592 2.88 5.32 34
B.3 A Slide No. Valence Arousal Slide No. Valence Arousal 1740 8.02 5.53 2703 2.33 5.73 2340 7.65 5.35 3000 1.69 6.74 4002 7.69 7.15 3001 2.1 5.85 4007 7.7 7.39 3015 1.83 5.54 4008 7.75 6.94 3053 1.5 6.2 4141 7.46 6.73 3069 2.1 6.7 4225 7.57 6.94 3071 2.06 6.61 4664 7.99 7.72 3080 1.63 6.84 5700 7.7 5.94 3100 1.88 5.88 7230 7.42 5.81 3170 1.77 6.79 7350 7.08 5.4 3195 2.56 6.23 7405 7.08 6.03 3266 1.98 5.85 7508 7 5.1 6313 2.43 6.54 8380 7.25 6.02 6520 2.45 5.85 7380 2.61 5.53 9040 1.88 5.1 9322 2.64 5.8 9921 2.6 6.09 35
B.4 B Slide No. Valence Arousal Slide No. Valence Arousal 1710 8.02 5.53 2811 2.84 6.31 4008 7.75 6.94 3000 1.69 6.74 4071 7.37 6.78 3001 2.1 5.85 4141 7.46 6.73 3015 1.83 5.54 4664 7.99 7.72 3053 1.5 6.2 4680 7.73 5.94 3064 1.78 5.44 5260 7.47 6 3069 2.1 6.7 5621 7.28 6.96 3071 2.06 6.61 5700 7.7 5.94 3080 1.63 6.84 7405 7.08 6.03 3100 1.88 5.88 7502 7.3 5.74 3170 1.77 6.79 8420 7.61 5.71 3195 2.56 6.23 3266 1.98 5.85 6230 2.73 7.1 6313 2.43 6.54 6520 2.45 5.85 7380 2.61 5.53 9040 1.88 5.1 9322 2.64 5.8 9410 1.96 6.38 36
B.5 C Slide No. Valence Arousal Slide No. Valence Arousal 1710 8.02 5.31 1052 2.99 6.89 2155 7.17 5.56 1201 2.93 6.87 2209 7.95 5.91 1271 2.65 6.01 2224 7.63 5.06 1274 2.86 5.66 2347 8.35 5.88 3000 1.21 7.77 4614 7.71 5.38 3103 1.71 6.6 5825 8.02 5.58 3150 1.98 6.94 7508 7.03 5.06 3170 1.2 7.55 8190 8.08 6.16 3213 2.61 6.79 8499 7.7 5.56 6311 2.36 5.12 6350 1.44 7.52 6560 1.78 6.86 9295 2.31 5.28 9326 1.84 6.28 37
B.6 D Slide No. Valence Arousal Slide No. Valence Arousal 1463 7.81 5.11 1271 2.65 6.01 1710 8.59 5.31 1274 2.86 5.66 2155 7.17 5.56 2095 1.48 5.72 2347 8.35 5.88 2800 1.41 5.87 4575 7.61 6.12 3000 1.21 7.77 4641 7.21 5.33 3103 1.71 6.6 5629 7.15 6.52 3150 1.98 6.94 5825 8.02 5.58 3170 1.2 7.55 5833 8.27 5.14 3213 2.61 6.79 8190 8.08 6.16 3400 2.06 7.12 6311 2.36 5.12 9181 1.98 6.09 9253 1.6 5.65 9326 1.84 6.28 38
B.7 E Slide No. Valence Arousal Slide No. Valence Arousal 1710 8.02 5.53 3000 1.21 7.77 2340 7.65 5.35 3053 1.5 6.2 4002 7.69 7.15 3064 1.78 5.44 4008 7.75 6.94 3069 2.1 6.7 4141 7.46 6.73 3071 2.06 6.61 4664 7.99 7.72 3100 1.88 5.88 4680 7.73 5.94 3170 1.77 6.79 5700 7.7 5.94 3266 1.98 5.85 7502 7.3 5.74 6415 2.81 5.86 8030 7.29 7.32 7380 2.61 5.53 8190 8.13 6.41 9414 2.43 6.36 8380 7.25 6.02 8420 7.61 5.71 39
C C 4.1 4.2 SPM5 p 0.001 t Statistics 40
C.1 A p1 41
C.2 A p2 42
C.3 B 43
C.4 B 44
C.5 C p1 45
C.6 C p2 46
C.7 C p3 47
C.8 C p1 48
C.9 C p2 49
C.10 C p3 50
C.11 C p4 51
C.12 D 52
C.13 D p1 53
C.14 D p2 54
C.15 D p3 55
C.16 E 56
C.17 E p1 57
C.18 E p2 58