1 1 1 Augmented Telepresence Using Recorded Aerial Omnidirectional Videos Captured from Unmanned Airship Fumio Okura 1, Masayuki Kanbara 1 and Naoka

Size: px

Start display at page:

Download "*1 1 1 Augmented Telepresence Using Recorded Aerial Omnidirectional Videos Captured from Unmanned Airship Fumio Okura *1, Masayuki Kanbara 1 and Naoka"

らむなかきむら
5 years ago
Views:

1 *1 1 1 Augmented Telepresence Using Recorded Aerial Omnidirectional Videos Captured from Unmanned Airship Fumio Okura *1, Masayuki Kanbara 1 and Naokazu Yokoya 1 Abstract This paper proposes an augmented telepresence system which generates large-scale omnidirectional AR scenes in outdoor environments using recorded omnidirectional videos captured from an unmanned airship. The proposed system realizes geometric registration by estimating extrinsic camera parameters using a video sequence and GPS data. Photometric registration is realized by environmental map completion and illumination environment estimation of real scene. This paper describes a system for aerial imaging using an airship and an omnidirectional camera, geometric and photometric registration methods, and experimental results of generating AR scenes from an omnidirectional video sequence captured on a historic site. Keywords : Augmented telepresence, Aerial image, Unmanned airship, Geometric registration, Photometric registration 1 [1] [1] [2] [3] Aspen MovieMap [4] Cybercity Walker [5] [6] Google Google Street View Augmented Reality: AR [7, 8] [9 12] Augmented Telepresence [10, 11] *1 *1 Nara Institute of Science and Technology (NAIST) CG 1 GIS [13, 14]

2 Vol.16, No.2, AR [ 1.1] [ 1.1] GPS [15] Structurefrom-Motion SfM SfM GPS [ 1.2] [ 1.2] Image Based Lighting (IBL) GI [ 2.1] 処理 1 処理 2 処理 3 処理 4 処理 5 全方位動画像の空撮カメラ位置姿勢推定全方位動画像の姿勢統一死角のない全方位動画像の生成 AR シーンのレンダリングアプローチ 2.1 アプローチ 1.1 アプローチ 2.3 アプローチ 2.2 アプローチ Fig. 1 Flow of proposed system [ 2.1] [ 2.2] [ 2.2] [ 2.3] [ 2.3] GPS 3

3 : 4 5 GI AR AR 2.3 [16] 2(a) 12m Ladybug3 Point Grey Research GPS Differential GPS P4-GPS TISS (b) 3 [17] Table 1 Specification of airship 12 m 3.2 m 12 kg 50km / h 60 GPS 姿勢センサ : TISS-5-40( 東京計器 ) (a) GPS アンテナノート PC,GPS 本体姿勢センサ全方位カメラ (b) 全方位カメラ : Ladybug 3(Point Grey Research) GPS アンテナ :P4-GPS( 日立造船 ) は飛行船上部に取付 2 Fig. 2 Unmanned airship and equipped sensors 1 PC 2.4 Structure-from-Motion SfM GPS [15] SfM SfM GPS GPS [15] [15] 4 SfM 2 Table 2 Specification of sensors Point Grey Research 15 fps Ladybug3 : 6 GPS Differential GPS P4-GPS 4Hz RMS 40.4cm RMS 53.5cm : 3 /h TISS Hz

4 Vol.16, No.2, 2011 ω S pi i F g GPS 3 Fig. 3 Omnidirectional image captured from airship GPS 測地座標系 GPS 測位値 g GPS 測位誤差 GPS 受信機カメラパラメータ行列カメラ座標系 11 GPS 受信機の推定位置投影座標 GPS 受信機の位置推定誤差 Ψ 特徴点再投影誤差 Φ 特徴点検出座標 4 [15] Fig. 4 Coordinates in camera pose estimation [15] i j q ij j ˆq ij i j Φ ij Φ ij = q ij ˆq ij 2. (1) GPS i GPS GPS g i M i GPS GPS d M i g i = d. (2) M i GPS Ψ i Ψ i = d M i g i 2. (3) Φ ij Ψ i E E = ω (1 ω) Ψ i + F g i F g i S Φ ij. (4) p i i j S pi M i ω E GPS GPS d GPS GPS GPS [15] GPS RTK-GPS Differential GPS RTK- GPS cm cm Differential GPS [15] ω = 10 9 ω = GPS [15] GPS E [15] 3 1 1

: 3. 5 Fig. 5 Omnidirectional image aligned its posture 2.5 2.4 i M i R i R 1 i = R T i GPS 3 5 5 2.6 2.5 6(a) 5 1. 2. 1 2.

5 : 3. 5 Fig. 5 Omnidirectional image aligned its posture i M i R i R 1 i = R T i GPS (a) Sum of Squared Difference (SSD) 1 1 6(b) 1 7(a) 1 v xy i v ixy 1 v xy = α ixy v ixy, (5) i α ixy i { 1 (v ixy ) α ixy = 0 (v ixy ) 7(b) i α ixy = 0 7(b)

[18] All Sky Model High Dynamic Range(HDR) All Sky Model All

Lz(γ s ) ϕ( π 2 ) f( π 2 γ s), (6) b ϕ(γ) = 1 + a exp( sin γ

73 a = 1 + 0.17 exp(3.7 Si) 1.16, 0.566 b = {1 + 30.1 exp( 2.

8, 3.05 d = 1 + 10.6 exp( 3.4 Si), 0.48 e = 1 + 245 exp( 4.

6 Vol.16, No.2, 2011 (a) (b) 6 Fig. 6 Completion of ground area (a) (b) (c) All Sky Model (d) HDR 7(a) 7 Fig. 7 Completion of sky area All Sky Model [18] All Sky Model [18] All Sky Model High Dynamic Range(HDR) All Sky Model All Sky Model La(γ s, γ, ζ) = Lz(γ s ) L(γ s, γ, ζ) ϕ(γ) f(ζ) = Lz(γ s ) ϕ( π 2 ) f( π 2 γ s), (6) b ϕ(γ) = 1 + a exp( sin γ ), f(ζ) = 1 + c {exp(d ζ) exp(d π 2 )}+e cos2 ζ, 6.73 a = exp(3.7 Si) 1.16, b = { exp( 2.8 Si)} 0.88, c = 1.77 (1.22 Si) 3.56 exp(0.2 Si) (2.1 Si) 0.8, 3.05 d = exp( 3.4 Si), 0.48 e = exp( 4.13 Si), La(γ s, γ, ζ) :, Lz(γ s ) :, L(γ s, γ, ζ) :, γ s : [rad], γ : [rad], ζ : [rad], Si : Si 0.0 Si 2.0 Si γ s All Sky Model La Lz [W/m 2 sr]

7 : 曇天晴天 Si = 0.0 Si = 1.0 Si = 2.0 Si = 0.6 (Lz の分散最小 ) 8 Fig. 8 All Sky Model Si Estimation of Si in All Sky Model RGB Lz Lz (6) La L Lz Lz(γ s, γ, ζ, Si) = La(γ s, γ, ζ) L(γ s, γ, ζ, Si). (7) Si Si k La k k L k Si Si = 0.0 Si = L k (γ s, γ, ζ, Si) (7) Lz k (γ s, γ, ζ, Si) 8 Lz k (γ s, γ, ζ, Si) Si Si Lz k (γ s, γ, ζ, Si) k Lz(γ s, γ, ζ, Si) Si Si Si Lz (6) Fig. 9 Result of environmental map completion AR AR Autodesk 3ds Max mental image mentalray IBL GI AR GPS m 600m 11 1km 1.3km 3.2 AR 12 12(a) 12(b) 12(c) AR 12(d) All Sky Model Si Si = 0.6 [19]

Vol.16, No.2, 2011 緯度 34.6935 34.693 34.6925 34.692 34.6915 34.691 シーケンス終点シーケンス始点 34.

8 Vol.16, No.2, 2011 緯度シーケンス終点シーケンス始点経度 10 Fig. 10 Path of airship during aerial imaging above heijo-kyo 11 Fig. 11 3D model of Heijo-kyo created by Toppan printing Co., Ltd. AR TV HMD AR [19] GPU HMD TV 14(a) 65 TV HMD 14(b) HMD HMD 3 HMD GPS

9 : (a) (b) (c) (d) AR 12 AR Fig. 12 Omnidirectional AR scene generated by proposed system

10 Vol.16, No.2, AR Fig. 13 Planar perspective AR scene generated from omnidirectional AR scene (a) TV (b) HMD 14 Fig. 14 Proposed system implemented for demonstration

11 : 再投影誤差 (Deg) フレーム提案システムセンサのみ 15 Fig. 15 Reprojection errors based on position and posture of the camera p < 0.01 t A B 3.3 HMD 5 1: 5: A 2.1 B 4.4 p < 0.01 Wilcoxon 4 GPS AR AR HMD TV ( (A) No ) 1300 [1] J. Steuer. Defining virtual reality: Dimensions determining telepresence. J. of Communication, Vol. 42, No. 4, pp , [2] G.Ballantyne. Robotic surgery, telerobotic surgery, telepresence, and telementoring. Surgical Endoscopy, Vol. 16, No. 10, pp , [3] T. Fong and C. Thorpe. Vehicle teleoperation interfaces. Autonomous Robots, Vol. 11, pp. 9 18, [4] A. Lippman. Movie-maps: An application of the optical videodisc to computer graphics. Proc. ACM SIGGRAPH 80, Vol. 14, No. 3, pp , [5] D. Kotake, T. Endo, F. Pighin, A. Katayama, H. Tamura, and M. Hirose. Cybercity walker 2001: Walking through and looking around a realistic cyberspace reconstructed from the physical world. In Proc. Second IEEE and ACM Int. Symp. on Augmented Reality, pp , [6] M. Hori, M. Kanbara, and N. Yokoya. MR telepresence system with inertial force sensation using a motion platform and an immersive display. In Proc. IEEE Sympo. on 3D User Interfaces (3DUI2009), pp , 2009.

Vol.16, No.2, 2011 [7] R. Azuma. A survey of augmented reality. Presence: Teleoperators and Virtual Environments, Vol. 6, No. 4, pp. 355 385, 1997. [8] R. Azuma, Y. Baillot, R. Behringer, S.

Parker. Augmented reality as a tool to aid the telerobotic exploration and characterization of remote environments. Presence: Teleoperators and Virtual Environments, Vol. 2, No. 4, pp. 352 567, 2002.

Applying telepresence and augmented reality to teleoperate field robots. J. of Robotics and Autonomous Systems, Vol. 26, No. 2-3, pp. 117 125, 1999. [12] F. Okura, M. Kanbara, and N. Yokoya.

12 Vol.16, No.2, 2011 [7] R. Azuma. A survey of augmented reality. Presence: Teleoperators and Virtual Environments, Vol. 6, No. 4, pp , [8] R. Azuma, Y. Baillot, R. Behringer, S. Feiner, S. Julier, and B. MacIntyre. Recent advances in augmented reality. IEEE Computer Graphics and Applications, Vol. 21, No. 6, pp , [9] S. Lawson, J. Pretlove, A. Weeler, and G. Parker. Augmented reality as a tool to aid the telerobotic exploration and characterization of remote environments. Presence: Teleoperators and Virtual Environments, Vol. 2, No. 4, pp , [10] H. Kim, J. Kim, and S. Park. A bird s-eye view system using augmented reality. In Proc. 32nd Annual Simulation Symposium, pp , [11] A. Halme, J. Suomela, and M. Savela. Applying telepresence and augmented reality to teleoperate field robots. J. of Robotics and Autonomous Systems, Vol. 26, No. 2-3, pp , [12] F. Okura, M. Kanbara, and N. Yokoya. Augmented telepresence using autopilot airship and omni-directional camera. In Proc. Ninth IEEE Int. Sympo. on Mixed and Augmented Reality (IS- MAR2010), pp , [13],,,. CG. :, Vol. 42, No. SIG6(CVIM2), pp , [14] P. Ghadirian and I. D. Bishop. Integration of augmented reality and GIS: A new approach to realistic landscape visualisation. Landscape and Urban Planning, Vol. 86, pp , [15] Y. Yokochi, S. Ikeda, T. Sato, and N. Yokoya. Extrinsic camera parameter estimation based-on feature tracking and GPS data. In Proc. Seventh Asian Conf. on Computer Vision (ACCV2006), Vol. 1, pp , [16] E. Paiva, J. Azinheira, J. Ramos, A. Moutinho, and S. Bueno. Project AURORA: Infrastructure and flight control experiments for a robotic airship. J. of Field Robotics, Vol. 23, No. 2 3, pp , [17] S. Ikeda, T. Sato, and N. Yokoya. High-resolution panoramic movie generation from video streams acquired by an omnidirectional multi-camera system. In Proc IEEE Int. Conf. on Multisensor Fusion and Integration for Intelligent System (MFI2003), pp , [18] N. Igawa, Y. Koga, T. Matsuzawa, and H. Nakamura. Models of sky radiance distribution and sky luminance distribution. Solar Energy, Vol. 77, pp , [19] Y. Onoe, K. Yamazawa, H. Takemura, and N. Yokoya. Telepresence by real-time viewdependent image generation from omnidirectional video streams. Computer Vision and Image Understanding, Vol. 71, No. 2, pp , ( ) IEEE ( ) 2002 FIT2005 IEEE IEEE ACM SIGGRAPH

& Vol.5 No (Oct. 2015) TV 1,2,a) , Augmented TV TV AR Augmented Reality 3DCG TV Estimation of TV Screen Position and Ro

& Vol.5 No (Oct. 2015) TV 1,2,a) , Augmented TV TV AR Augmented Reality 3DCG TV Estimation of TV Screen Position and Ro TV 1,2,a) 1 2 2015 1 26, 2015 5 21 Augmented TV TV AR Augmented Reality 3DCG TV Estimation of TV Screen Position and Rotation Using Mobile Device Hiroyuki Kawakita 1,2,a) Toshio Nakagawa 1 Makoto Sato