( )

Size: px

Start display at page:

Download "( )"

りえひがき
5 years ago
Views:

1 NAIST-IS-MT Visual SLAM

2 ( )

3 Visual SLAM Visual SLAM (Simultaneous Localization and Mapping) Visual SLAM SfM(Structure from Motion) Visual SLAM Visual SLAM, NAIST-IS-MT , i

4 ( ) Visual SLAM,, Visual SLAM,, ii

5 Cumulative Error Reduction Using Aerial Images in Visual SLAM for Ground-View Video Takuya Miyamoto Abstract Visual Simultaneous Localization and Mapping (SLAM) methods have been proposed for Augmented Reality (AR) applications and car navigation systems. Generally, visual SLAM often suffers from cumulative errors when taking a long video sequence for a wide area. For this problem, cumulative errors have been reduced by loop closing, which corrects camera poses from estimated 3D environments when camera returns to a previously observed location while taking a video. However, this method cannot be applied when we do not take the same scene at least twice. A conventional study of Structure from Motion (SfM) for offline processing corrects a camera poses on the basis of feature matching between a ground-view video and external references, e.g. like aerial images. However, since the computational cost of the method is high, it is difficult to apply the method to applications that requires online processing. To solve this problem, on the basis of a SLAM method based on feature points, this thesis proposes a camera pose estimation method that achieves both online processing and reduction of cumulative errors by using correspondences of edges between the ground-view video and aerial images. To make correspondences between a ground-view video and aerial images, the proposed method detects a Master s Thesis, Graduate School of Information Science, Nara Institute of Science and Technology, NAIST-IS-MT , March 10, iii

6 ground surface from 3D points estimated by a SLAM method based on feature points, and transforms each key-frame in the ground-view video to a front-parallel rectified view (which is referred to as an air-view image). The proposed method then estimates a camera pose by minimizing both re-projection errors in a feature point based SLAM method and distances of edges detected from air-view and aerial images. Using edges for making correspondences between a ground-view video and aerial images suppresses calculation cost and the proposed method consequently achieves online processing. Experiments demonstrate the effectiveness of the proposed method by examining the estimation accuracy of camera poses. Keywords: camera pose estimation, cumulative error, Visual SLAM, air-view image iv

7 Visual SLAM Visual SLAM Visual SLAM ICP v

8 39 40 vi

9 1 NAIST SfM [1] Visual SLAM Drummond [2] Taketomi [3] Kume [4] ( ) vii

10 viii

11 ix

12 1. 3 Visual SLAM (Simultaneous Localization and Mapping) [5 12] Visual SLAM 3 [13, 14] 2 GPS [15 17] 3 [2,3,18 21] [4,22 27] GPS [15 17] GPS GPS GPS GPS GPS 3 [2, 3, 18 21] [4, 22 27] Visual SLAM 3 1

13 Visual SLAM Visual SLAM Visual SLAM

14 [28] GPS [29 31] Newman [28] Feiner [29] Gleue [30] Piekarski [31] GPS 2.2 GPS 3D 3

15 1: NAIST SfM [1] SfM(Structure from Motion) Visual SLAM SfM [1,32] 1 3 ( 3 2 ) [33,34] Visual SLAM 3 [5 12] Visual SLAM 2(a) (feature based method) [5 8] 2(b) (direct method) [9 12] [5 8] 4

(a) 特徴点に基づく手法 [5] (b) 画素値に基づく手法 [10] 図 2: Visual SLAM の一例を追跡することでカメラ位置姿勢を推定する一方画素値に基づく手法 [9 12] は各画素の Photo-consistency が最大となるようにカメラ位置姿勢を推定するこれらの手法は

たマップを利用してカメラ位置姿勢を補正するループクロージングと呼ばれる手法 [13, 14] が提案されているが同一環境を 2

16 (a) 特徴点に基づく手法 [5] (b) 画素値に基づく手法 [10] 図 2: Visual SLAM の一例を追跡することでカメラ位置姿勢を推定する一方画素値に基づく手法 [9 12] は各画素の Photo-consistency が最大となるようにカメラ位置姿勢を推定するこれらの手法は少数のキーフレームに対してのみバンドル調整を行うため計算コストは低いしかしながら広域な環境を対象として長時間カメラ位置姿勢を行った場合カメラ位置姿勢の誤差が蓄積するという問題があるこの蓄積誤差を軽減するために一度撮影した地点を再度観測した際にこれまでに構築したマップを利用してカメラ位置姿勢を補正するループクロージングと呼ばれる手法 [13, 14] が提案されているが同一環境を 2 回以上観測しない場合には適用することができない外部指標を用いる手法動画像と外部指標を併用する手法においては外部指標として GPS [15 17] 3 次元モデル [2, 3, 18 21] 航空写真 [4, 22 27] などを用いることで蓄積誤差を解消する手法が研究されている GPS を用いる手法 GPS を用いる手法 [15 17] として再投影誤差とカメラの推定位置に対する GPS の測位位置の誤差との和で定義されたエネルギー関数を最小化することで動画全体のカメラ位置姿勢を推定する拡張バンドル調整と呼ばれる手法が提案されているこれらの手法では絶対的なカメラ位置を推定できるが GPS の測位 5

17 GPS GPS 3 3 [2, 18] 3 [3, 19 21] Drummond [2] 3 Bleser [18] SLAM 3 Fioraio [19] 3 SLAM SLAM Lothe [20] GIS Geographic Information System 3 SLAM GIS 3 ICP(Iterate Closest 6

3: Drummond [2] Point) SLAM 3 GIS 3 Tamaazousti [21] 3 3

18 3: Drummond [2] Point) SLAM 3 GIS 3 Tamaazousti [21] Taketomi [3] 4 SfM 3 [26,27] [4,22 25] Leung [27] 7

19 4: Taketomi [3] 3 Kim [26] GPS IMU(Inertial Measurement Sensor) 3 3 Pink [23] SfM 8

20 Toriya [22] GPS SIFT Bansal [24] Noda [25] Kume [4] SIFT SIFT SfM Visual SLAM SfM Visual SLAM GPS 3 GPS GPS 9

図 5: Kume らによる上空視点画像と航空写真の対応付け [4] 表 1: 従来のカメラ位置姿勢の推定手法の特徴カメラ位置姿勢の推定手法推定精度蓄積誤差オンライン処理ユーザによる外部指標の準備コストセンサベース低いなしあり動画像 (SfM) 高いありなし動画像 (Visual SLAM) 高いありあり動画像+GPS GPS の精度に依存なしあり無し

21 図 5: Kume らによる上空視点画像と航空写真の対応付け [4] 表 1: 従来のカメラ位置姿勢の推定手法の特徴カメラ位置姿勢の推定手法推定精度蓄積誤差オンライン処理ユーザによる外部指標の準備コストセンサベース低いなしあり動画像 (SfM) 高いありなし動画像 (Visual SLAM) 高いありあり動画像+GPS GPS の精度に依存なしあり無し動画像+3 次元モデル高いなしあり高い動画像+航空写真高いなしなし低い精度が大きく低下する問題および GPS の測位結果が長時間取得できない区間において GPS の測位情報を推定結果に反映することが難しいという問題がある 3 次元モデルを用いた手法では広範囲な屋外環境における 3 次元モデルの作成やデータベースの構築にかかる人的コストが大きいという問題がある航空写真を用いる手法では既に構築されている航空写真データベースから航空写真を容易に入手できるため 3 次元モデルを用いる場合に比べて環境を新たに計測する必要がないという利点があるしかし従来手法は動画像全体を対象とした一括での最適化を行うことを想定しており Visual SLAM のようにオンライン処理で逐次出力が要求されるアプリケーションに適用することは困難であるこれら 10

22 3 Visual SLAM Visual SLAM Visual SLAM 11

23 3. Visual SLAM Visual SLAM Visual SLAM Visual SLAM Mapping Thread Visual SLAM 3 2 ICP 3.2 Visual SLAM PTAM Visual SLAM 6 Tracking Thread Mapping Thread Tracking Thread Mapping Thread Mapping 12

24 6: Thread 3 3 Visual SLAM 3.3 Visual SLAM 2 ICP 13

25 3.3.1 Visual SLAM 3 RANSAC [35] (1) (4) 3 (1) Visual SLAM 3 3 (2) 3 (3) 3 (4) (1) (3) C RANSAC 3 3 N (u C,v C ) (u A,v A ) λu C u A λv C = K λ C M W toc M 1 v A W toa K 1 A (1) λ K C K A M W toc M W toa 14

26 M W toc = M W toa = K C = K A = ( XC 0 Y C 0 Z C 0 T C 1 ) (2) ( XA Y A Z A ) T A (3) f x 0 c xc 0 0 f y c yc 0 (4) s 0 0 c xa 0 s 0 c ya (5) (f x,f y ) (c xc,c yc ) s (c xa,c ya ) 7 (X A Y A Z A ) (X C Y C Z C ) T A T C (X A Y A Z A ) 7 X A = Z C Z A Z C Z A (6) Y A = X A Z A X A Z A Z A = N N 3 15 (7) (8)

27 7: T =(t x, t y, t z ) T ={M W toa M 1 W toc ( ) T } T (1) (2) (3) t z = 1 n n i=1 p z i t x = t z(c xa c x ) f xc t y = n xt x +n z t z +d d = (n x 1 n n y n i=1 p x i + n y 1 n n i=1 p y i + n z 1 n n i=1 p z i ) (9)

28 M AtoM 8 ( ) s θ (t x t y ) 3 3 E edge (M AtoM ) = M AtoM = { N Tn ( b i M AtoM m i > T n ) i=1 b i M AtoM m i 2 scos(θ) ssin(θ) t x ssin(θ) scos(θ) t y otherwise (10) (11) N m i i b i m i Visual SLAM M AtoM canny [36] [37] canny 9 17

29 8: () ICP (11) 4 2 ICP [38] (10) (11) ICP Visual SLAM (1) (4) (1) s θ (t x,t y ) M AtoM (2) M AtoM m M AtoM m 18

30 (a) (b) (c) 9: (3) b i (2) m i (4) (10) s θ (t x,t y ) (5) (10) (2) Visual SLAM Mapping Thread E 19

31 10: E({M W toci } I i=1, {p j } J ) = E rep ({M W toci } I i=1, {p j } J j=1) +λe edge ({M W toci } I i=1, {p k } K k=1) (12) I J 3 K p j j 3 p k b ak 3 E rep 3 20

32 2 I J E rep ({M W toci } I i=1, {p j } J j=1) = m ij v(i, p j ) 2 (13) ( ) λv(i, pj ) 1 i=1 j=1 = K Ci M W toci p j (14) m ij i p j E edge E edge ({M W toci } I i=1, {p ak } K k=1) = + K b k M W tom p ak 2 k=1 K m h(k)j v(h(k), p ak ) 2 (15) k=1 h k M W tom M W tom = M K A M CtoA M W toc (16) ( ) M MAtoM 0 = (17) 0 1 (10) 3 3 Sparse Bundle Adjustment [39] 3 21

33 ( 12) ICP Visual SLAM ATAM [6] ATAM 11 8 ICP 2 3 Zhang [40] 22

図 11: 本実験で使用した動画像に対するカメラ位置の真値赤線と提案手法で航空写真と位置合わせするキーフレームの撮影地点黄色表 2: 動画像を撮影したカメラの仕様と内部パラメータ使用したカメラ GoPro Hero3+ 解像度 848x480 水平画角 122.6 垂直画角 94.

34 図 11: 本実験で使用した動画像に対するカメラ位置の真値赤線と提案手法で航空写真と位置合わせするキーフレームの撮影地点黄色表 2: 動画像を撮影したカメラの仕様と内部パラメータ使用したカメラ GoPro Hero3+ 解像度 848x480 水平画角垂直画角 94.4 フレームレート [fps] 240 内部パラメータ (fx,fy ) (381,384) 画像中心 (cx,cy ) (420,239) 表 3: 提案手法で用いるパラメータ RANSAC の繰り返し回数 (cnt) 2000 平面から 3 次元点群までの距離 0.1 最近傍点の探索の距離 5.0 上空視点画像の解像度 500x500 上空視点画像の直交投影パラメータ (s,cxa,cya ) (30,250,250) 23

35 (a) 地点 1 (b) 地点 2 (c) 地点 3 (d) 地点 4 (e) 地点 5 (f) 地点 6 (g) 地点 7 (h) 地点 8 図 12: 本実験で外部指標を与えたキーフレーム画像 24

36 4.2 Visual SLAM ATAM [6] (a) 11 13(b)

37 (a) (b) 13: 3 26

38 (b) 21(b) RANSAC 20(c)

39 (a) (b) (c) 3 14: 1 (a) (b) (c) 3 15: 2 28

40 (a) (b) (c) 3 16: 3 (a) (b) (c) 3 17: 4 (a) (b) (c) 3 18: 5 29

41 (a) (b) (c) 3 19: 6 (a) (b) (c) 3 20: 7 (a) (b) (c) 3 21: 8 30

42 ICP 31

43 (a) (b) (c) 22: 1 (a) (b) (c) 23: 2 32

44 (a) (b) (c) 24: 3 (a) (b) (c) 25: 4 33

45 (a) (b) (c) 26: 5 (a) (b) (c) 27: 6 34

46 (a) (b) (c) 28: 7 (a) (b) (c) 29: 8 35

47 PC

48 30: 31: 37

49 5. Visual SLAM 3 3 Visual SLAM 38

50 39

51 [1] C. Wu. A Visual Structure from Motion System [2] Tom Drummond and Roberto Cipolla. Real-time visual tracking of complex structures. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 24, No. 7, pp , [3] Takafumi Taketomi, Tomokazu Sato, and Naokazu Yokoya. Real-time and Accurate Extrinsic Camera Parameter Estimation using Feature Landmark Database for Augmented Reality. Int. Journal of Computers & Graphics, Vol. 35, No. 4, pp , [4] Hideyuki Kume, Tomokazu Sato, and Naokazu Yokoya. Bundle adjustment using aerial images with two-stage geometric verification. Computer Vision and Image Understanding, Vol. 138, pp , [5] Georg Klein and David Murray. Parallel Tracking and Mapping for Small AR Workspaces. In Proc. IEEE and ACM Int. Symp. on Mixed and Augmented Reality, pp , [6] Hideaki Uchiyama, Takafumi Taketomi, Sei Ikeda, Silva Do Monte Lima, and Joao Paulo. Abecedary Tracking and Mapping: A Toolkit for Tracking Competitions. In Proc. IEEE and ACM Int. Symp. on Mixed and Augmented Reality, pp , [7] Chieh-Chih Wang, Charles Thorpe, Sebastian Thrun, Martial Hebert, and Hugh Durrant-Whyte. Simultaneous Localization, Mapping and Moving Object Tracking. The Int. Journal of Robotics Research, Vol. 26, No. 9, pp , [8] Andrew J Davison, Ian D Reid, Nicholas D Molton, and Olivier Stasse. MonoSLAM: Real-time Single Camera SLAM. IEEE Trans. on Pattern 40

52 Analysis and Machine Intelligence, Vol. 29, No. 6, pp , [9] Richard A Newcombe, Steven J Lovegrove, and Andrew J Davison. DTAM: Dense Tracking and Mapping in Real-Time. In Proc. Int. Conf. on Computer Vision, pp , [10] Jakob Engel, Thomas Schöps, and Daniel Cremers. LSD-SLAM: Large-Scale Direct Monocular SLAM. In Proc. European Conf. on Computer Vision, pp [11] Jakob Engel, Jurgen Sturm, and Daniel Cremers. Semi-Dense Visual Odometry for a Monocular Camera. In Proc. IEEE and ACM Int. Symp. on Mixed and Augmented Reality, pp , [12] Christian Kerl, Jurgen Sturm, and Daniel Cremers. Dense Visual SLAM for RGB-D Cameras. In Proc. IEEE and RSJ Int. Conf. on Intelligent Robots and Systems, pp , [13] Paul Newman and Kin Ho. SLAM-Loop Closing with Visually Salient Features. In Proc. Int. Conf. on Robotics and Automation, pp , [14] Adrien Angeli, Stéphane Doncieux, Jean-Arcady Meyer, and David Filliat. Visual topological SLAM and global localization. In Proc. Int. Conf. on Robotics and Automation, pp , [15] Maxime Lhuillier. Incremental Fusion of Structure-from-Motion and GPS using Constrained Bundle Adjustments. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 34, No. 12, pp , [16],,,,,. GPS., Vol. 43, No. 1, pp ,

53 [17],,,. GPS.., Vol. 47, No. 5, pp , [18] Gabriele Bleser, Harald Wuest, and D Strieker. Online camera pose estimation in partially known and dynamic scenes. In Proc. IEEE and ACM Int. Symp. on Mixed and Augmented Reality, pp , [19] Nicola Fioraio and Luigi Di Stefano. Joint Detection, Tracking and Mapping by Semantic Bundle Adjustment. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition, pp , [20] Pierre Lothe, Steve Bourgeois, Eric Royer, Michel Dhome, and Sylvie Naudet-Collette. Real-time Vehicle Global Localisation with a Single Camera in Dense Urban Areas: Exploitation of Coarse 3D City Models. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition, pp , [21] Mohamed Tamaazousti, Vincent Gay-Bellile, Sylvie Naudet Collette, Steve Bourgeois, and Michel Dhome. Nonlinear refinement of structure from motion reconstruction by taking advantage of a partial knowledge of the environment. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition, pp , [22] Hisatoshi Toriya, Itaru Kitahara, and Yoshichika Ohta. A Mobile Camera Localization Method Using Aerial-View Images. In The 2nd Asian Conf. on Pattern Recognition, pp , [23] Oliver Pink, Frank Moosmann, and Alexander Bachmann. Visual Features for Vehicle Localization and Ego-Motion Estimation. In Proc. IEEE Intelligent Vehicles Symp., pp ,

54 [24] Mayank Bansal, Kostas Daniilidis, and Harpreet Sawhney. Ultra-wide Baseline Facade Matching for Geo-Localization. In Proc. European Conf. on Computer Vision, pp , [25] Masafumi Noda, Tomokazu Takahashi, Daisuke Deguchi, Ichiro Ide, Hiroshi Murase, Yoshiko Kojima, and Takashi Naito. Vehicle Ego-localization by Matching In-vehicle Camera Images to an Aerial Image. In Proc. Computer Vision in Vehicle Technology, pp , [26] Sehwan Kim, Stephen DiVerdi, Jae Sik Chang, Taehyuk Kang, Ronald Iltis, and Tobias Höllerer. Implicit 3D Modeling and Tracking for Anywhere Augmentation. In Proc. ACM Symp. on Virtual Reality Software and Technology, pp , [27] Keith Yu Kit Leung, Christopher M Clark, and Jan P Huissoon. Localization in Urban Environments by Matching Ground Level Video Images with an Aerial Image. In Proc. IEEE Int. Conf. on Robotics and Automation, pp , [28] Joseph Newman, David Ingram, and Andy Hopper. Augmented reality in a wide area sentient environment. In Proc. IEEE and ACM Int. Symp. on Augmented Reality, pp , [29] Steven Feiner, Blair MacIntyre, Tobias Höllerer, and Anthony Webster. A Touring Machine: Prototyping 3D Mobile Augmented Reality Systems for Exploring the Urban Environment. Proc. IEEE Int. Symp. Wearable Computers, Vol. 1, No. 4, pp , [30] Tim Gleue and Patrick Dähne. Design and Implementation of a Mobile Device for Outdoor Augmented Reality in the Archeoguide Project. In Proc. Conf. Virtual Reality, Archeology, and Cultural Heritage, pp , [31] Wayne Piekarski, David Hepworth, Victor Demczuk, Bruce Thomas, and Bernard Gunther. A Mobile Augmented Reality User Interface for Terrestrial 43

55 Navigation. In Proc. Australasian Computer Science Conf., pp , [32] Noah Snavely, Steven M Seitz, and Richard Szeliski. Modeling the World from Internet Photo Collections. Proc. Int. Journal of Computer Vision, Vol. 80, No. 2, pp , [33] Bill Triggs, Philip F McLauchlan, Richard I Hartley, and Andrew W Fitzgibbon. Bundle Adjustment A Modern Synthesis. In Proc. Int. Workshop on Vision algorithms, pp [34] Marc Pollefeys, Luc Van Gool, Maarten Vergauwen, Frank Verbiest, Kurt Cornelis, Jan Tops, and Reinhard Koch. Visual Modeling with a Hand-held Camera. Int. Journal of Computer Vision, Vol. 59, No. 3, pp , [35] Martin A. Fischler and Robert C. Bolles. Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography. Comm. of the ACM, Vol. 24, No. 6, pp , [36] John Canny. A Computational Approach to Edge Detection. IEEE Trans. on Pattern Analysis and Machine Intelligence, No. 6, pp , [37] W. M. Wells, III. Efficient Synthesis of Gaussian Filters by Cascaded Uniform Filters. IEEE Trans. on Pattern Analysis and Machine Intelligence, No. 8(2), pp , [38] Szymon Rusinkiewicz and Marc Levoy. Efficient Variants of the ICP Algorithm. In Proc. Third Int. Conf. on 3-D Digital Imaging and Modeling, pp , [39] Kurt Konolige and Willow Garage. Sparse Sparse Bundle Adjustment. In Proc. of the British Machine Vision Conference, pp. 1 11,

56 [40] Zhengyou Zhang. A Flexible New Technique for Camera Calibration. IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol. 22, No. 11, pp ,

(bundle adjustment) 8),9) ),6),7) GPS 8),9) GPS GPS 8) GPS GPS GPS GPS Anai 9) GPS GPS GPS GPS GPS GPS GPS Maier ) GPS GPS Anai 9) GPS GPS M GPS M inf

(bundle adjustment) 8),9) ),6),7) GPS 8),9) GPS GPS 8) GPS GPS GPS GPS Anai 9) GPS GPS GPS GPS GPS GPS GPS Maier ) GPS GPS Anai 9) GPS GPS M GPS M inf GPS GPS solve this problem, we propose ()novel model about GPS positioning which enables more robust estimation with extended bundle adjustment, and ()outlier removal for GPS positioning using video information.