修士論文の和文要旨 研究科 専攻 大学院情報理工学研究科情報 通信工学専攻博士前期課程 氏名大和田瑛美華学籍番号 1431024 論文題目 MixedReality 空間における仮想書斎システムの開発 要 旨 本研究では,MR(MixedReality, 複合現実 ) 空間を用いた仮想書斎システムを開発した. 本システム内では, 電子書籍の閲覧を行ったり, 壁に写真を飾ったりすることが出来る. また, ユーザ自身が本当に歩くことによって仮想空間内を移動したり, 手で触る操作をすることでページをめくったり物を移動したりすることが出来る. 近年電子書籍が普及しているものの, ページの厚みにより読んだ量や残りの量が分かり易い ページを捲 ( めく ) ることで読んでいる実感がある など紙書籍ならではの利点もある. そこで本研究では, 電子書籍でありながら紙書籍に近い感覚が得られることを目標とした. 本システムは, ユーザーが歩き回れる仮想空間を構築すること, ユーザーの手の動きに対して仮想空間内で反応させること, 紙書籍に近い形状の電子書籍をモデリング レンダリングすること, の三つの機能に分けられる.Kinect を用いてユーザーの位置を, 基準マーカーを用いて顔の向きを検出し, 仮想空間内での移動を実現している. ユーザーの手の動きは Kinect で検出し, 仮想空間内での座標系と照らし合わせることで対応付け, ページ捲りや物の移動を実現している.Kinect で取得出来る関節のデータには時々大きなノイズが生じるため, ユーザーデータを元に補正方法を開発した. 読み進めた分のページを OpenGL を用いて三次元的に表現し, ページが捲れる様子も立体的にページを動かすことで, 読んでいる感覚を得やすい様に表現した. 本システムと Kindle for PC を用いて読後の記憶に関する比較実験を行った結果, 一つの課題の正答率は本システムが 27%,Kindle が 20% で若干優れている数字が出たが, もう一つの課題ではほぼ差が見られず, 有意な差とは言えなかった. 被験者の意見では, 本システムの方が読んでいる楽しさがあるという意見もあったが, 操作性や見た目などに対する課題も多かった.
27 MixedReality 1431024 28 3 15
MR MixedReality Kinect Kinect Kinect OpenGL Kindle for PC 27% Kindle 20%
1 3 1.1....................................... 3 1.2....................................... 3 1.3....................................... 4 2 6 2.1........................... 6 2.2.................................. 8 2.2.1 Kinect v2[3]............................... 8 2.2.2 LeapMotion[4].............................. 8 3 12 3.1......................... 12 3.2 VS............................... 12 4 14 4.1 MR..................................... 14 4.1.1 MR Plug-in for Unity : MREAL[22].................. 14 4.1.2..................... 14 4.2............................. 17 4.2.1 [25]............... 17 4.2.2..................... 17 4.3 VR..................................... 17 4.3.1 Virtuix Omni[26]............................ 17 4.3.2..................... 18 4.4............................ 18 4.4.1 Variable Reality II[27]......................... 18 4.4.2 Kindle[28]................................ 21 5 23 5.1........................... 23 5.1.1............................ 23 5.1.2.................... 23 5.1.3.................. 24 5.2............................ 26 5.2.1................................. 26 5.2.2 MR............................ 26 5.2.3......................... 26 5.2.4................................. 27 1
6 28 6.1.............................. 28 6.1.1.............................. 28 6.1.2............................. 28 6.2................................. 31 6.2.1............................. 31 6.2.2.................... 32 6.2.3 (................... 35 6.2.4..................... 38 6.2.5......................... 39 6.2.6............................. 43 7 44 7.1........................ 44 7.1.1................................... 44 7.2 LeapMotion Kinect........ 44 7.2.1..................... 44 7.2.2................................. 45 7.2.3 EC2015.......................... 45 7.3 Kinect...................... 45 7.3.1..................... 45 7.3.2................................. 46 7.4........................ 46 7.4.1................................... 46 7.4.2............................. 46 7.4.3................................. 48 8 52 8.1...................................... 52 8.1.1 MR......................... 52 8.1.2 MR....................... 53 8.1.3 MR....................... 53 8.1.4..................... 54 8.1.5..................... 54 9 58 2
1 1.1 CG Computer Graphics,Oculus Rift[12] GoogleGlass[2] HMD Head Mount Display),HMD,Kinect[3],LeapMotion[4] VR Virtual Reality HMD VRHMD,, PR,,, HMD 10 2013 [18] 1.2, MR, MR, 3
1.1: 1.3 1 2 VR AR MR 3 4 5 6 7 LeapMotion Kinect Kinect 4
8 5
2 VR AR MR 2.1 VR Virtual Reality AR Augmented Reality MR Mixed Reality ) VR ( 2.1) ( 2.2) AR CG ( 2.3, 2.4) MR AR VR AR MR PR 2.1: VR [5] 2.2: VR ( [6] ) 6
2.3: AR AR[7] 2.4: AR ( [8] ) 7
2.2 2.2.1 Kinect v2[3] Microsoft 3 Kinect v2( Kinect ( 2.5 Kinect. Xbox360 DanceEvolution Kinect 2.1 6 Depth ) Depth Time of Flight(TOF:Kinect Depth Kinect 2.2 Kinect Color 2.6,Depth 2.7, 2.8 Color Depth Kinect 2.2.2 LeapMotion[4] LeapMotion( 2.9), LeapMotion 150fps 120fps KeyTap (Circle Kinect,Oculus Rift[12] LeapMotion LeapMotion ( 2.10), ( 2.11) Oculus Rift HMD 8
2.1: Kinect v2 [9] OS Windows 8,Windows 8.1, Windows Embedded Standard 8, Windows Embedded Standard 8.1 36W CPU 2.66 GHz 64 (x64) USB 3.0, PCI-Express USB PCI-EX Gen2 (5Mbps) 2 GB RAM GPU DirectX 11 Visual Studio 2012 Visual Studio 2013.NET Framework 4.5 2.2: Kinect v2 [10] Color :1920 1080, fps:30fps Depth( ) :1920 1080,fps:30fps, :0.5m 8.0m Player 6 Skeleton( 2 Joint 25 Hand State Angle 70 60 2.5: Kinect v2 9
2.6: Color 2.7: Depth 2.8: Kinect Body) 10
2.9: LeapMotion 2.10: Oculus Rift LeapMotion [11] Leap Motion VR Gallery 2.11: Oculus Rift LeapMotion [11] Planetarium 11
3 3.1,2013 [13] CD-ROM :1980 1990 :1990 2000 ):2000 2000 :2000 2010 :2010 1985 CD- ROM [14] 2009 9 2010 Kindle ( ) iphone ipad 2015 2 [15][16] 50 [17] 3.2 VS Stavanger Anne Mangen, [19],, 12
[20][21],,,, 1,, 2010,, 13
4 4.1 MR 4.1.1 MR Plug-in for Unity : MREAL[22] MR IT MREAL Unity D MREAL MR HM-A1 ( 4.1) MR ( 4.2) 4.1.2 [22] MR MREAL 4.3 4.1: MREAL [24] ) 14
4.2: MREAL 2 [23] ) 4.3: MREAL [24] ) 15
16
4.4: AR LeapMotion ([25] ) 4.2 4.2.1 [25] CG AR MR AR CG Kinect(Kinect v1 LeapMotion CG Kinect LeapMotion LeapMotion 4.4 LeapMotion LeapMotion 4.2.2,, 4.3 VR 4.3.1 Virtuix Omni[26] VR, Virtuix Virtuix Omni 4.5 VR,,, 17
4.5: Virtuix Omni [26] PRODUCTS ) Omni, Omni Shoes Omni Tracking POD,Omni,Omni Omni, Omni Omni 4.6 4.3.2,,, 4.4 4.4.1 Variable Reality II[27] HYE SOO YANG Variable Reality II LeapMotion Oculus Rift AR VR 4.7 18
4.6: Virtuix Omni [26] ATX VR Meetup November 2015 ) 4.7: Variable Reality [27] Variable Reality Setup ) 4.8: AR [27] Variable Reality Augmented Book ) 19
4.9: AR [27] Variable Reality Demonstration Video ) 4.10: VR [27] Variable Reality Virtual Book ) 4.11: VR [27] Variable Reality Demonstration Video ) 20
4.12: Kindle for PC 4.13: Kindle for PC PDF AR ( 4.8 4.9) VR ( 4.10 4.11) VR AR 4.4.2 Kindle[28] Amazon.com Kindle Amazon.com PC 4.12 4.13 Windows8.1\8\7 Kindle for PC[29] Kindle for PC Enter 21
Kindle for PC PC PC 22
5 5.1 2 5.1.1 5.1.2 Kindle SonyReaderStore[30] PDF 1 23
5.1.3 MR 1 MR 24
MR MR MR CG 25
5.2 Kinect LeapMotion PC Web HMD Kinect LeapMotion 5.2.1 Kinect PC Kinect LeapMotion Web USB HMD HDMI 5.2.2 MR Kinect X Y HMD MR MR LeapMotion Kinect LeapMotion MR Kinect MR 5.2.3 2 26
5.2.4 Kinect 27
6 6.1 Kinect Windows8 Visual Studio( VS) C++ 6.1.1 Kinect LeapMoiton HMD Web PC Kinect v2 SDK v2.0 1409 LeapMotion SDK 2.3.1 2015 12, Kinect for Windows v2,xbox One Kinect v2 Windows PC,PC HMD:HMZ-T3 HMD Sony 3D HMZ-T3 [31] PC HDMI 3D 6.1.2 freeglut OpenGL OpenGL freeglut[32] freeglut GLUT the OpenGL Utility Toolkit), freeglut 2.8.1 OpenCV OpenCV(Open Sourse Computer Vision Library)[33] GUI( 28
6.1: ARToolkit 6.2: ARToolkit OpenCV2.4.8 OpenCV OpenCV3.0 OpenCV OpenCV HP[33] OpenCV [34] VS VS Cmake(ver.3.3.0-rc2) VS2015 VS2013 OpenCV :ARUco[35] OpenCV (opencv contrib) ARUco ARUco Web ARToolkit[36] OpenGL Kinect v2 OpenCV RGB Color ARUco ARToolkit hiro ( 6.1 ) 6.1 6.2 ARUco ARUco 6.3 ID 29
6.3: AR [35] 30
6.4: 6.2 6.5: MR 6.4 ARUco PC PC HMD ARUco HMD MR,, 6.5 MR 6.2.1 VS Web Kinect LeapMotion 31
6.6: Kinect MR ARUco ARUco 6.2.2 Kinect 6.6 Kinect Color Depth PC Color X Depth Z Kinect Depth 6.7 6.7 Kinect Kinect MR Kinect 2m 3, Kinect, 3, 6.8 Color, 6.9 6.10 Depth, 32
6.7: Kinect 6.8: Color 6.9: 33
6.10: 34
6.11: 6.2.3 ( Kinect SDK HMD ARUco 6.11 HMD ARUco Kinect ARUco,,, 4,,, Kinect Color ARUco,Color,ARUco Kinect Color, 3 1 HMD Kinect HMD 6.12 Kinect 6.13 6.14 35
6.12: ( [37] ) 36
6.13: 6.14: Kinect 37
6.15: LeapMotion HMD HMD LeapMotion ( 6.15) LeapMotion Kinect Kinect 4 ARUco 6.2.4 Kinect MR X Y Z Kinect Kinect LeapMotion Kinect LeapMotion PC OpenGL, 38
6.16: LeapMotion MR LeapMotion SDK,MR 6.16 LeapMotion Web OpenCV Web Web Web LeapMotion,Web LeapMotion LeapMotion 6.2.5 6.5 6.17 6.18 6.19 Web HMD OpenGL OpenGL OBB Oriented Bounding Box) OBB 39
6.17: 6.18: 40
6.19: 6.20: OBB 41
6.21: 6.22: OBB 6.20 OBB X Y Z 6.21 OBB2 6.21 6.22 OBB OBB OBB OBB 42
6.2.6 6.23: MR 6.23 MR 43
7 LeapMotion Kinect Kinect 7.1 3 Kinect 2 3 (JointType Head), (JoitType SpinShoulder) 3 (3Frame SpinShoulder) (3FrameBody) 7.1 7.1.1 3 7.2 LeapMotion Kinect 7.2.1 2015 9 25 ( ) 27 ( ) 2015(EC2015) 9 26 HMD LeapMotion Kinect EC2015 Kinect MR LeapMotion MR 44
7.2.2 CG LeapMotion LeapMotion 25cm LeapMotion LeapMotion 7.2.3 EC2015 LeapMotion LeapMotion 2 Leap- Motion LeapMotion LeapMotion 7.3 Kinect 7.3.1 2015 11 22 Kinect Web EC2015 LeapMotion Kinect MR 45
7.3.2 10 MR ( ) Kinect 7.2 7.3 (JoitType Head) getdeptcoodinateaverage Z) get3framedepthcoodinatebodyaverage 7.4 7.4.1 Kindle for PC[29] Kindle for PC HMD PC 7.4.2 1. (a) HMD Web PC (b) Kinect (c) (d) 3 46
7.1: 7.2: 7.3: 47
7.1: [%] 26.67 62.00 Kindle for PC 20.00 64.00 (e) 1 2. Kindle for PC (a) 3 (b) 1 3. 1 1 10 20 2 1 3, 3 6 4,Kinect 1 7.4.3 5 Kindle for PC 7.1 - HMD - HMD, HMD, 48
-, PC,,, - MR,, -, -, ( ), 49
- - HMD Web HDMI USB,,, -,,,,,, -, - Kindle for PC Kindle, HMD HMD 50
( 7.1) 51
8 8.1 Kinect,, MR,, 8.1.1 MR Kinect, MR,Kinect Depth,, MR -, -,, 52
8.1.2 MR MR,OBB, LeapMotion Kinect MR Kinect LeapMotion Kinect LeapMotion Kinect 8.1.3 MR MR Kinect, -,,MR -, x z, 53
8.1.4 - - - - 8.1.5 54
[1] Oculus Rift Oculus VR, LLC https://www.oculus.com [2] Google Glass Google https://developers.google.com/glass/ [3] Xbox 360,Microsoft http://www.xbox.com/ja-jp/kinect [4] LeapMotion https://www.leapmotion.com/ [5] The London Heist Getaway Sony Morpheus VR E3 2015,2016 1 28 https://www.youtube.com/watch?v=tmi2swxc1em [6] VR,2016 1 28 http://www.cigr.co.jp/pj/hako/ [7] AR,2016 1 28 http://www.morinaga.co.jp/kyorochan/ar/ [8] AR GooglePlay,2016 1 28 https://play.google.com/store/apps/details?id=jp.co.morinaga. kyorochanar [9] Xbox One Kinect Windows PC,2016 1 28 http://www.microsoftstore.com/store/msjp/ja_jp/pdp/xbox-one-kinect/ productid.309515600 [10] Kinect v1 Kinect v2,2016 1 28 http://www.buildinsider.net/small/kinectv2cpp/01 [11] Setup Leap Motion for VR Development,2016 1 28 https://developer.leapmotion.com/vr-setup [12] Oculus,2016 1 28 https://www.oculus.com/ [13], 2013 ) http://www.jftc.go.jp/cprc/reports/index.files/cr-0113.pdf 55
[14] 2014, https://r.impressrd.jp/iil/files/sample_pdf/ 2014\_.pdf [15] 2015 5 14 http://research.lifemedia.jp/ [16] Voxburner 62% of 16-24s prefer books as physical products 2015 5 14 http://www.voxburner.com/blog-source/2015/5/18/ 16-24-prefer-books-as-physical-products [17] ( ) 2015 5 14 http://www7b.biglobe.ne.jp/~yama88/info.html [18] 2014 ) [19] Want to Remember What You Read? Switch to Paper,2016 1 28 http://www.inc.com/laura-montini/paper-might-help-your-ability-to-recall- what-you-read.html [20] 9 vs.,2016 1 28 http://ddnavi.com/news/55944/a/ [21] http://mind.c.u-tokyo.ac.jp/sakai_lab_files/staff/kls_paperj/ KLS2013JNd.pdf [22] MREAL IT 2013,2016 1 28 http://www.canon-its.co.jp/company/news/20130618mr01.html [23] MR Plug-in for Unity IT 2013 http://www.canon-its.co.jp/mr/pdf/mrpunity_catalog_1306.pdf [24] MREAL IT,2016 1 28 http://web.canon.jp/v-square/movie.html?id=t025 [25] 2013) [26] Virtuix Omni,Virtuix,2016 1 28 http://www.virtuix.com/ [27] Variable Reality II,HYE SOO YANG,2016 1 28 http://www.hyesooy.com/varreality2 56
[28] Kindle,2016 1 28 https://www.amazon.co.jp/kindle- - /b/ref=topnav_ storetab_kinc?ie=utf8&node=2250738051 [29] Kindle for PC,2016 1 28 http://www.amazon.co.jp/gp/feature.html?docid=3078592246 [30] SonyReaderStore,2016 1 31 http://ebookstore.sony.jp/ [31] Personal 3D Viewer,Sony,2016 1 28 http://www.sony.jp/hmd/products/hmz-t3/ [32] freeglut,2016 1 28 http://freeglut.sourceforge.net/ [33] OpenCV,2016 1 28 http://opencv.org/ [34] OpenCV GitHub,2016 1 28 https://github.com/itseez [35] Automatic generation and detection of highly reliable fiducial markers under occlusion S. Garrido-Jurado, R. Munoz-Salinas, F.J Madrid-Cuevas, M.J. Mar n-jim enez, http://www.uco.es/investiga/grupos/ava/sites/default/files/ GarridoJurado2014.pdf [36] ARToolkit HLTL(Human Interface Technology),2016 1 28 http://www.hitl.washington.edu/artoolkit/ [37] UNITY-CHAN! OFFICIAL WEBSITE,2016 1 28 http://unity-chan.com/ 57
9,,,,,, 58