1,a) 1 1 (Spatial Augmented Reality SAR) 1000 fps ms SAR SAR Low-latency Spatial Augmented Reality Based on High-speed Projector and Wearable Type Tracking Hikaru Amano 1,a) Yoshihiro Watanabe 1 Masatoshi Ishikawa 1 Abstract: Spatial augmented reality is useful technique due to project information corresponding to the environment. However, with existing systems, it was difficult to realize a system that interacts with people because the system has delay of projection and sensing technologys. When the observer perceives the delay of the image, the observer feels a sense of incompatibility and motion sickness. Therefore, it is important to present images without delay. In this research, by incorporating a high-speed projector and high-speed image sensing, we show the possibility to construct the system that throughput is 1000 fps and latency is millisec order. Additionally, in such applications, we need to caputre the human viewpoint and motion with high accuracy and high speed in a wide space. Therefore, in this paper, we propose a configuration combining a wearable high-speed vision and a high-speed projector due to capture the observer viewpoint. 1. (Spatial Augmented Reality SAR) [1] CAVE [2] 1 Graduate School of Information Science and Technology, University of Tokyo a) hikaru amano@ipc.i.u-tokyo.ac.jp SAR [3], [4] Ng 6.04 ms 210
1 Fig. 1 We displayed the schedule seen in static state in sight. The observer and the image are moving together for making the image seen in a static state. The image captured with wearable camera [5] Head Mounted Display (HMD) [7] [8] SAR 1 RoomAlive [3] 144 fps 7 ms 530 fps 5.43 ms 1 2. 2.1 Spatial Augumented Reality SAR Illumiroom [4] Invoked computing [6] RoomAlive [3] ms 2.2 1 [9], [10], [11]. 2.3 [12]. 211
(c) 2 (c) Fig. 2 The whole system. a high-speed projector and a wearable vision and (c) a screen 3 Fig. 3 19 15 DDCM DDCM(19 15) currently used. [13], [14] 3. 3.1 1000 fps ms SAR SAR 2 ms DynaFlash [15] 3 ms 8 bit 1000 fps 3.2 [14], [16] 3.3 3.2 ms 1 ms Deformable Dot Cluster Marker (DDCM) [8] DDCM DDCM 3 212
1000fps 0.8 m 1.1 m 30fps Fig. 5 5 The whole system at the time of experiment 4 1000 fps 30 fps 30 fps 3 Fig. 4 Projection image taken with 30 fps vision and 1000fps vision. detection frame-by-frame tracking 2 detection ID frame-by-frame tracking [13] 700 fps 75% 1000 fps 3 1000 fps 3 4 3 30 fps 4 1 3.3 1 X = (x, y, z, 1) T U = (u, v, 1) T P 6 Fig. 6 Tracking success rate 1.2 1 0.8 0.6 0.4 0.2 0 0 100 200 300 Distance between screen and camera(cm) The upper figure is that system tracking accuracy at varying distances between vision and screen. and are images of the marker taken by the visions at each point. U PX 2 3 2 DDCM 3 8 DDCM 4 213
Image cap Fig. 8 marker tracking Track ing Image cap Processing times 8 transformation transfers to the projector and projects images 0.6 2.43 5.43 Processing time(ms) transform 7 Fig. 7 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 Track ing Image cap transfer to the projector transform Track ing projects images transfer to the projector transform Latency and throughput 0 10 20 30 40 50 Number of tracking point projects images transfer to the projector ms ms projects images Tracking processing times at varying tracking success points. 4. 5 1.1 0.8 m 1024 768 pixel 1000 fps 3 basler aca-750uc 320 240 pixel 1000 fps 6 mm 19 15 ( 3) GPU CPU CPU Intel Xeon CPU E5-2687W v4 @ 3.00GHz 2 6 5 cm 10000 4 120 cm 180 cm DDCM 6 6 6 7 1 8 0.6 ms 2 1000 1.83 ms CPU [8] GPU 1 ms GPU 1 ms 3.0 ms 3.0 ms [15] 5.43 ms 7 1.83 ms 530 fps 5. 3.3 3.3 (c) 214
(c) (d) 9 (c) (d) Fig. 9 A state of transformation when the observer moves. Moving left and right. Moving up and down. (c) Moving back and forward. (d) Rotate to left and right. (d) 1.1 0.8 m Head-Up Display (HUD) 6. 5.43 ms SAR DDCM [1] Bimber, Oliver, and Ramesh Raskar. Spatial augmented reality: merging real and virtual worlds. CRC press, 2005. [2] Cruz-Neira, Carolina, et al. The CAVE: audio visual experience automatic virtual environment. Communications of the ACM 35.6 1992: 64-73. [3] Jones, Brett, et al. RoomAlive: magical experiences enabled by scalable, adaptive projector-camera units. Proceedings of the 27th annual ACM symposium on User interface software and technology. ACM, 2014. [4] Jones, Brett, et al. IllumiRoom: peripheral projected illusions for interactive experiences. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2013. [5] Ng, Albert, et al. Designing for low-latency direct-touch input. Proceedings of the 25th annual ACM symposium on User interface software and technology. ACM, 2012. [6] Zerroug, Alexis, et al. Invoked computing: Spatial audio and video AR invoked through miming Proceedings of Virtual Reality International Conference. 2011. [7] Lincoln, Peter, et al. From Motion to Photons in 80 Microseconds: Towards Minimal Latency for Virtual and Augmented Reality. IEEE transactions on visualization and computer graphics 22.4 2016: 1367-1376. [8] Narita, Gaku, et al. Dynamic Projection Mapping onto Deforming Non-rigid Surface using Deformable Dot Cluster Marker. IEEE Transactions on Visualization and Computer Graphics 2016. [9] ( ). 17.3 2012: 219-229. [10] Shiratori, Takaaki, et al. Motion capture from bodymounted cameras. ACM Transactions on Graphics (TOG). Vol. 30. No. 4. ACM, 2011. [11] Rhodin, Helge, et al. EgoCap: egocentric marker-less motion capture with two fisheye cameras. ACM Transactions on Graphics (TOG). Vol. 35. No. 6. ACM, 2016. [12] Asayama, Hirotaka, et al. Diminishable visual markers on fabricated projection object for dynamic spatial augmented reality. SIGGRAPH Asia 2015 Emerging Technologies. ACM, 2015. [13] (, ).. MVE, 113.109 2013: 71-76. [14] Niidome, Hidetaka, et al. Camera synchronization to imperceptible frames embedded in a displayed video sequence. SIGGRAPH Asia 2014 Posters. ACM, 2014. [15] Watanabe, Yoshihiro, et al. High-speed 8-bit image projector at 1,000 fps with 3 ms delay. Proceedings of the International Display Workshops. 2015. [16] Goto, Akifumi, et al. Display tracking using blended images with unknown mixing ratio as a template. SIG- GRAPH ASIA 2016 Technical Briefs. ACM, 2016. 215