NAIST-IS-MT

Transcription

1 NAIST-IS-MT

2 ( )

3 Augmented Reality AR AR AR AR AR (1) (2) (3) AR AR, NAIST-IS-MT , i

4 AR AR AR ii

5 Augmented Reality Using Pre-captured Images Considering Change of Real-world Illumination Takayuki Akaguma Abstract Augmented Reality (AR) using mobile devices (e.g., smartphones and tablets) can be used for various applications such as the visualization of lost buildings at cultural heritage sites. Recently, a novel type of mobile AR, a pre-captured AR system, has been proposed. In the pre-captured AR, omnidirectional images are captured at a number of fixed points and virtual objects are rendered with high quality in an offline process. In the online process, the system shows user perspective images generated from omnidirectional AR images based on the estimated pose of the mobile device. Unlike conventional AR systems that register CG and captured images in real time, the pre-captured AR system achieves: (1) geometric registration without any perceptible misalignment between captured images and virtual objects, (2) high-quality photometric registration by offline rendering for virtual objects, and (3) online process with low computational cost on the mobile device. One problem of naive pre-captured AR system is the reduction of the reality in an AR experience due to the difference in illumination environments between the pre-captured image and the real world, which is caused by the difference of the weather and light position between them. In this thesis, we propose a novel pre-captured AR system considering the change of real-world illumination. First, omnidirectional images are preliminarily Master s Thesis, Department of Information Science, Graduate School of Information Science, Nara Institute of Science and Technology, NAIST-IS-MT , March 13, iii

6 captured under various illumination environments. In the online-stage, one of pre-captured images that is similar with respect to the illumination of the scene to the image captured in real-time is then selected and is displayed to the user. In order to investigate the validity of our proposed system, we have carried out an experiment using an omnidirectional image database captured under a variety of weather and time. The result of the experiment has demonstrated that the proposed system can improve the reality in comparison with a conventional precaptured AR system, which does not consider real-world illumination. We also have developed an application for a virtual history experience in Todaiji, and have investigated the validity of the proposed method through the public experiment. Keywords: mobile augmented reality pre-captured AR geometric consistency, photometric consistency, omnidirectional image iv

7 AR AR AR AR RGB AR AR v

8 vi

9 1 Visual-SLAM [9] (PTAM on iphone [4]) [20] [22] AR AR AR [23] AR s 0.5 < s < 1.5, vii

10 22 Q Q2-Q Q3 Q (Q1) AR (Q2) (Q3) (Q4) AR (Q1) (Q2) (Q1) viii

11 1. (Augmented Reality: AR) AR AR GPS AR AR AR AR AR 1

12 AR AR 2 AR AR 3 AR 4 AR AR 5 2

13 2. AR AR 2.1 AR (Augmented Reality: AR) AR HMD AR AR [1] [2] AR AR

14 2.1.1 AR [2] [3] [4] [5] Layer 1 Wikitude 2 Junaio 3 [4] [5] Visual-SLAM(Simulataneous Localization and Mapping) [4] [6] [7] 1 Klein [4] Klein [4] KLT [8] Visual-SLAM 2 Visual-SLAM

15 (a) (b) 1 Visual-SLAM [9] SLAM [10] [11] 5

16 2 (PTAM on iphone [4]) [12] Lepetit [12] [13] Structure from Motion

17 [14] Taketomi [5] [13] [15] Fournier [16] 7

18 3 4 [20] [17] Kinect [18] 3 [19] 4 [20] [21] GPU 8

19 (a) 86.8 (b) 5 5 [22] Csongei [22] AR AR Wither [23] AR AR 9

20 6 AR AR AR [23] AR AR AR 20 6 AR 20 6 [23] 7 AR AR AR AR [24] [25] 10

21 7 AR AR Wither [23] 10% AR AR AR 11

22 2.3 AR AR 2.2 Wither [23] AR AR (1) 9 (2) (3)

23 1 AR 13

24 8 9 14

25 10 15

26 3. AR 3.1 AR RGB RGB RGB AR 11 AR 16

27 AR [26] [27] 14 17

28

29 AR Autodesk 3ds Max IBL [28] AR

30 RGB RGB RGB RGB 16 RGB

31 3.3 RGB RGB s 18 ID i RGB H db i θ RGB H mob θ θ i 19 s θ diff c(i, θ, s) c (R, G, B) 21

32 s=1.0 s=

33 18 θ = 1 θ = 2 θ = 12 s = 0.5 s = 0.6 s = 0.7 s = 1.5 diff (i, 1, 0.5) diff (i, 2, 0.5) diff (i, 12, 0.5) diff (i, 1, 0.6) diff (i, 2, 0.6) diff (i, 12, 0.6) diff (i, 1, 0.7) diff (i, 2, 0.7) diff (i, 12, 0.7) diff (i, 1, 1.5) diff (i, 2, 1.5) diff (i, 12, 1.5) D c (i) s 0.5 < s < 1.5,

34 D c (i) D c (i) = min s diff c (i, θ, s). (1) θ s s diff c (i, θ, s) s s s θ diff c(i, θ, s) s 19 diff c (i, θ, s) diff c (i, θ, s) = 1 m(θ, s) m(θ,s) j=0 ( H mob θ (j) n mob ) 2 (sj) (2) n db (i, θ) Hdb i n mob n db (i, θ) m s 1 m = (Hθ mob ) s < 1 m = ( s (Hθ mob ) ) s m RGB D c (i) i D(i)

35 25

36 AR AR KMD a 1 20 b 2 ipad2, ipad mini and ipad 4th generation (Apple Inc.) Ladybug3 (Point Grey Research Inc.) 3ds Max (Autodesk, Inc.) KMD SYMPO2013/ 26

37 20 a GPS m

38 (a) (b)

39 15:00 17: Q1) Q2) CG Q3) Q4) Q1 Q2 Q3 Q Q AR Q2 Q4 23 Q2 Q3 7 29

40 Q % 50% 100% 22 Q1 Q Q3 Q4 0% 50% 100% 23 Q2-Q4 2 3m 3 Q4 4 30

41 4.2 AR ( 24(a)) ipad ( 24(b)) ( 24(c)) ipad

42 (a) (b) (c) 24 32

43 25 (1) (3) (2) m 7 Q1) Q2) 33

44 (a) Q3 (b) Q4 26 Q3 Q4 Q3) CG CG 26(a) Q4) 26(b) Q1 7 Q (1) (3) 7 Q3 Q (1) (3)

45 4.2.3 Q % 5,6 94% Q2 Q Q2 28 Q3 29 Q

46 27 (Q1) p < AR (Q2) 36

47 p < (Q3) 7 6 p < (Q4) 37

48 AR ipad ipad m

49

50 7 Q1) Q2) Q1 Q2 Q1 7 Q Q1 Q

51 35 Q Q1 36(a) 37(d) 36(e) 37(c) AR AR RGB AR AR 41

52 7 6 p < AR (Q1) 7 6 p < (Q2) 42

53 35 (Q1) 43

54 (a) (b) (c) (d) (e)

55 (a) (b) (c) (d) (e)

56 (a) (b) (c) (d)

57 (a) (b) (c) (d) (e)

58 (a) (b) (c) (d) (e)

59 5. AR AR 3 1 (1) (2) (3) 3 AR AR AR AR AR 49

60 50

61 NAIST KMD 51

62 [1] M Zoellner, J Keil, T Drevensek, and H Wuest. Cultural Heritage Layers: Integrating Historic Media in Augmented Reality. In Proc. Int l Conf. on Virtual Systems and Multimedia (VSMM 09), pp , [2] G A Lee and M Billinghurst. CityViewAR : A Mobile Outdoor AR Application for City Visualization. In Proc. IEEE Int l Symp. on Mixed and Augmented Reality (ISMAR 12) - Arts, Media, and Humanities Proceedings, pp , [3] R Azuma. A Survey of Augmented Reality. Presence: Teleoperators and Virtual Environments, Vol. 6, No. 4, pp , [4] G Klein and D Murray. Parallel Tracking and Mapping on a Camera Phone. In Proc. IEEE Int l Symp. on Mixed and Augmented Reality (ISMAR 09), pp , [5] T Taketomi, T Sato, and N Yokoya. Real-time and Accurate Extrinsic Camera Parameter Estimation Using Feature Landmark Database for Augmented Reality. Int l Journal of Computers and Graphics, Vol. 35, No. 4, pp , [6] E Eade and T Drummond. Scalable Monocular SLAM. In Proc. IEEE Computer Society Conf. on Computer Vision and Pattern Recognition (CVPR 06), pp , [7] A J Davison, W W Mayol, and D W Murray. Real-time Localisation and Mapping with Wearable Active Vision. In Proc. IEEE Int l Symp. on Mixed and Augmented Reality (ISMAR 03), pp , [8] J Shi and C Tomasi. Good Features to Track. In Proc. IEEE Computer Society Conf. on Computer Vision and Pattern Recognition (CVPR 94), pp ,

63 [9] G Klein and D Murray. Parallel Tracking and Mapping for Small AR Workspaces. In Proc. IEEE Int l Symp. on Mixed and Augmented Reality (ISMAR 07), pp , [10] H Kato and M Billinghurst. Marker Tracking and HMD Calibration for a Video-based Augmented Reality Conferencing System. In Proc. IEEE and ACM Int l Workshop on Augmented Reality (IWAR 99), pp , [11] L Naimark and E Foxlin. Circular Data Matrix Fiducial System and Robust Image Processing for a Wearable Vision-inertial Self-tracker. In Proc. IEEE Int l Symp. on Mixed and Augmented Reality (ISMAR 02), pp , [12] V Lepetit, L Vacchetti, D Thalmann, and P Fua. Fully Automated and Stable Registration for Augmented Reality Applications. In Proc. IEEE Int l Symp. on Mixed and Augmented Reality (ISMAR 02), pp , [13],,.., Vol. 10, No. 3, pp , [14],,.. D-II), Vol. J88-D-II, No. 2, pp , [15] H W Jensen. Global Illumination Using Photon Maps. In Proc. Eurographics Workshop on Rendering (EGWR 96), pp , [16] A Fournier, Atjeng S. G, and Chris R. Common Illumination between Real and Computer Generated Scenes. In Proc. Graphics Interface (GI 93), pp , [17] M F Cohen, S E Chen, J R Wallace, and D P Greenberg. A Progressive Refinement Approach to Fast Radiosity Image Generation. In Proc. ACM SIGGRAPH 88, pp ,

64 [18] R A Newcombe, A J Davison, S Izadi, P Kohli, O Hilliges, J Shotton, D Molyneaux, S Hodges, D Kim, and A Fitzgibbon. KinectFusion: Realtime Dense Surface Mapping and Tracking. In Proc. IEEE Int l Symp. on Mixed and Augmented Reality (ISMAR 11), pp , [19] L Gruber, T Richter-Trummer, and D Schmalstieg. Real-time Photometric Registration from Arbitrary Geometry. In Proc. IEEE Int l Symp. on Mixed and Augmented Reality (ISMAR 12), pp , [20] P Kán and H Kaufmann. High-quality Reflections, Refractions, and Caustics in Augmented Reality and Their Contribution to Visual Coherence. In Proc. IEEE Int l Symp. on Mixed and Augmented Reality (ISMAR 12), pp , [21] C B Madsen and M Nielsen. Towards Probe-less Augmented Reality. In Proc. Int l Conf. on Computer Graphics Theory and Applications (GRAPP 08), pp , [22] M Csongei, L Hoang, C Sandor, and Y Lee. Global Illumination for Augmented Reality on Mobile Phones. In Proc. IEEE Virtual Reality (VR 14), 2 pages, [23] J Wither, Y-T Tsai, and R Azuma. Indirect Augmented Reality. Int l Journal of Computers and Graphics, Vol. 35, No. 4, pp , [24] G Liestø l and A Morrison. Views, Alignment and I ncongruity in Indirect Augmented Reality. In Proc. IEEE Int l Symp. on Mixed and Augmented Reality (ISMAR 13), pp , [25] S Côté, P Trudel, M-A Desbiens, M Giguère, B Systems, and R Snyder. Live Mobile Panoramic High Accuracy Augmented Reality for Engineering and Construction. In Proc. Int l Conf. on Construction Applications of Virtual Reality (CONVR 13), pp ,

65 [26] I Arai, M Hori, N Kawai, Y Abe, M Ichikawa, Y Satonaka, T Nitta, H Fujii, M Mukai, and Others. Pano Umechika: A Crowded Underground City Panoramic View System. In Proc. Int l Symp. on Distributed Computing and Artificial Intelligence (DCAI 10), pp [27] N Kawai, T Sato, and N Yokoya. Image Inpainting Considering Brightness Change and Spatial Locality of Textures and Its Evaluation. In Proc. Pacific- Rim Symp. on Image and Video Technology (PSIVT 09), pp , [28] P Debevec. Rendering Synthetic Objects into Real Scenes: Bridging Traditional and Image-based Graphics with Global Illumination and High Dynamic Range Photography. In Proc. ACM SIGGRAPH 98, pp ,