3 1, 2, 2, 3, 4, 5, 3, 6 Aurora 3d-measurement from whole-sky time series image using fish-eye stereo camera Akira TAKEUCHI 1, Hiromitsu FUJII 2, Atsu

Size: px

Start display at page:

Download "3 1, 2, 2, 3, 4, 5, 3, 6 Aurora 3d-measurement from whole-sky time series image using fish-eye stereo camera Akira TAKEUCHI 1, Hiromitsu FUJII 2, Atsu"

てるえはぎにわ
4 years ago
Views:

1 3 1, 2, 2, 3, 4, 5, 3, 6 Aurora 3d-measurement from whole-sky time series image using fish-eye stereo camera Akira TAKEUCHI 1, Hiromitsu FUJII 2, Atsushi YAMASHITA 2, Masayuki TANAKA 3, Ryuho KATAOKA 4, Yoshizumi MIYOSHI 5, Masatoshi OKUTOMI 3 and Hajime ASAMA 6 1, 2, 6 Department of Precision Engineering, The University of Tokyo Hongo, Bunkyo-ku, Tokyo , Japan 3 Department of Mechanical and Control Engineering, Tokyo Institute of Technology Ookayama, Meguro-ku, Tokyo , Japan 4 National Institute of Polar Research 10-3 Midori-cho, Tachikawa-shi, Tokyo , Japan 5 Solar-Terrestrial Environment Laboratory, Nagoya University Furo-cho, Chikusa-ku, Nagoya , Japan Received 31 July 2015 Abstract Three-dimensional analysis of aurora is important for the research of solar wind and magnetic storms, because aurora reflects the relationship between solar wind and terrestrial magnetism. Therefore a method to reconstruct a threedimensional shape of aurora precisely is demanded. In this paper, a method to measure three-dimensional shape of auroa by using only two fish-eye cameras is proposed. Two fish-eye cameras were installed at Fairbanks, Alaska, U.S.A to get the time series images of aurora. The images photographed by the cameras are performed calibration by using the star to estimate the camera attitude. The dense feature points are detected from the images of aurora by using template matching though the image of aurora has few characteristic pattern. Moreover, a feature point tracking between two images continuing in time is performed to improve precision of the feature point detection. Then the three-dimensional coordinates of feature points are calculated by triangulation, aurora shape is measured and visualized. Key words : Aurora 3D-Measurement, Fish-eye stereo camera, Whole-sky image, Template matching, Feature point tracking No [DOI: /transjsme ] ( ) 4 ( ) 5 ( ) 6 of corresponding author: takeuchi@robot.t.u-tokyo.ac.jp

2 Fig. 1 Image pair for stereo measurement 1. GPS (1999) Störmer 2 (Störmer, 1915) (Stenbaek and Hallinan, 1979, Brown et al., 1976) Aso Tanaka (Aso et al., 1998, Tanaka et al., 2011) 7 TV 3 (Sharp, 1971, Nishimura et al., 2010) 3 3 3

3 Aurora image conversion Aurora area extraction by background subtraction Detection and tracking feature points by template matching N Time and spatial direction consistency is satisfied Y Detection and tracking feature points by SIFT Time and spatial direction consistency is satisfied Y Three dimensional measurement and visualization Fig. 2 Flow chart of the proposal method (Mori et al., 2013) 2 2 3(a) 3(a) X X Y Z Y X Z

4 Optical axis direction of the installed left camera R Z Optical axis direction of the installed left camera Optical axis direction of the installed right camera Left camera Right camera Rectified coordinate system Camera posture in rectified coordinate system Installed camera posture (a) The conversion to the rectified coordinate system (b) Rotation matrix Fig. 3 The conversion to the rectified coordinate system using the star positions Direction vector to the star in the rectified coordinate system Altitude Z Rectified coordinate system Right camera (, ) X (, ) Y Left camera Longitude direction Sea level β α Latitude direction Azimuthal angle to right camera from left camera Elevation angle to right camera from left camera (a) Direction vector to the star in the camera coordinate system Fig. 4 (b) Direction vector to the star in the rectified coordinate system Direction vector to the star N N n 1, n 2,..., n i N m 1, m 2,..., m i i = 1,2,...,Nn i = (x i,y i,z i ) T m i = (X i,y i,z i ) T 3(b) 3(b) R (1) n i = Rm i (1) (1) n i m i 4(a) 4(b) 4(a) x y z

5 i S i (C u,c v )S i (u i,v i ) S i r i S i 4(a) (x i,y i,z i ) (θ i,ϕ i ) r i (2) (3) (3) k r i = (u i C u ) 2 + (v i C v ) 2 (2) r i = 2k sin θ i (3) 2 (2) (3) 4(a) (4) (5) (θ i,ϕ i ) (5) θ i = 2sin 1 r i 2k (4) ϕ i = cos 1 u C u, ϕ i = sin 1 v C v r i r i (θ i,ϕ i ) (x i,y i,z i ) (6) (5) n i = x i y i sinθ i cosϕ i = sinθ i sinϕ i (6) z i cosθ i (3) k (3) (7) r i k = 2sin θ (7) i 2 (7) θ i = 90 r i (2007) 90 (7) θ i = 90 r i m i m i 4(b) 4(b) X Y Z XYZ m i S i GPS m i R N (1) S i m i S i n i R n T i Rm i R

6 (8) e (9) e R (8) 9 1 e = 1 1 N N i=1 ( n T i Rm i ) (8) R = argmine (9) R 2 (3) (3) 2 2 Benezeth (Benezeth et al., 2008) 5 5(a) 5(b) pixel Zero-mean Normalized Cross-Correlation: ZNCC Scale-Invariant Feature TransformSIFT (Lowe, 2004) 2 4 3

7 (a) Original image Fig. 5 Background subtraction (b) Foreground image ZNCC SIFT SIFT SIFT ZNCC (a) 2 5 3

8 (a) (b) Time: t (d) Time: t+ t Left camera Right camera t Fig. 6 Tracking method of the feature point t t+ t t+ t t+ t (, 2014, Fujii et al., 2014) SIFT SIFT t SIFT t+ t t+ t ZNCC ZNCC 6 6 (a)(b) t(c)(d) t+ t (a)(c) (b)(d) 6 1 t t A t B t 2 A t B t t+ t A t B t A t+ t B t+ t 3 t+ t t+ t A t+ t B t+ t 4 B t+ t B t+ t B t+ t

9 B t+ t SIFT SIFT t SIFT SIFT t+ t 1 t+ t t t+ t SIFT ω Z ω (1999) h 1 [km] h 2 [km].(x i,y i,z i ) Z θ i 10 tanθ i = x 2 i + y2 i z i 2 (11)(12) 0 x 2 i + y2 i z i tan ω 2 h 1 z i h 2 (12) NURBS 3 (10) (11) 3.

(a) Outward appearance of fisheye camera system (b) Inside of

8 The place where the cameras were installed 3 1 7 7 2 2 8.

3 t 10 2 PC PC 2014 11 21 1 02 30 t 6,000 pixel 4,000 pixel

10 (a) Outward appearance of fisheye camera system (b) Inside of fisheye camera system Fig. 7 Fisheye camera system Fig. 8 The place where the cameras were installed km 8 2 PC 1.3 t 10 2 PC PC t 6,000 pixel 4,000 pixel 3,200 pixel 3,200 pixel 3 2 t 9(a) 9(b) 9(c) 9(d) (c) 9(d) 10(a) 10(b) 100 pixel 100 pixel t 10(a) 10(b) 10(c) 10(d)

(a) Original image of left camera (b) Original image of right camera (a) Image plotted feature points of left camera (b) Image plotted feature points of right camera (c) Converted image of left

10 (d) Image of right camera after feature point tracking Results of feature point detection and tracking 10(c) 10(d) 11 11(a) 11(b) 11(a) 11(b) 11(c) 11(d) 11(c) 11(d) 11(e) 11(f) 11(e) 11(f) 11(c)

11 (a) Original image of left camera (b) Original image of right camera (a) Image plotted feature points of left camera (b) Image plotted feature points of right camera (c) Converted image of left camera Fig. 9 (d) Converted image of right camera Image conversion (c) Image of left camera after feature point tracking Fig. 10 (d) Image of right camera after feature point tracking Results of feature point detection and tracking 10(c) 10(d) 11 11(a) 11(b) 11(a) 11(b) 11(c) 11(d) 11(c) 11(d) 11(e) 11(f) 11(e) 11(f) 11(c) 11(d) SIFT SIFT 12 12(a) t SIFT 12(b) 12(a) SIFT t+1 12(c) 12(d) 12(a) 12(b) 12(e) 12(f) 12(c) 12(d) 12(c) 12(f) 12(c) 12(d) t t+ t SIFT 2 h 1 [km] h 2 [km] Kataoka (Kataoka et al., 2013) h 1 = 50h 2 = (a) 13(b) t 13(d) 13(e) t+ t 13(c) 13(f) t t+ t NURBS

12 (a) Image before feature point tracking by template matching (b) Image after feature point tracking by template matching (a) Image before feature point tracking by SIFT (b) Image after feature point tracking by SIFT (c) Enlarged image before feature point tracking by template matching (d) Enlarged image after feature point tracking by template matching (c) Enlarged image before feature point tracking by SIFT (d) Enlarged image after feature point tracking by SIFT Fig. 11 (e) Background of enlarged Image before feature point tracking by template matching (f) Background of enlarged Image after feature point tracking by template matching Tendency of feature point tracking by using template matching Fig. 12 (e) Background of enlarged image before feature point tracking by SIFT (f) Background of enlarged image after feature point tracking by SIFT Tendency of feature point tracking by using SIFT 4. 3 SIFT 3 (Kataoka et al., 2013) SIFT 1 t t+2 t t SIFT 14 1 SIFT

(a) Aurora shape at time t as seen from the ground (b) Aurora shape at

using NURBS (d) Aurora shape at time t+ t as seen from the ground (e)

the aurora area Time The number of the pixels The number of feature

proposed method detected by only SIFT t 159,549 27,147 1,048 (17.01 ) (0.

13 (a) Aurora shape at time t as seen from the ground (b) Aurora shape at time t as seen from the side (c) Aurora shape at time t visualized by using NURBS (d) Aurora shape at time t+ t as seen from the ground (e) Aurora shape at time t+ t as seen from the side (f) Aurora shape at time t+ t visualized by using NURBS Fig. 13 3D-shape of aurora Table 1 The ratio of the detected feature points to the aurora area Time The number of the pixels The number of feature points The number of feature points in the aurora area detected by the proposed method detected by only SIFT t 159,549 27,147 1,048 (17.01 ) (0.66 ) t+ t 136,540 31,479 1,078 (23.05 ) (0.79 ) t+2 t 167,812 32,486 1,242 (19.36 ) (0.74 ) SIFT t+2 t s t t+2 t

(a) Detected feature points in the left camera s image by only SIFT at time t (b) Detected feature points in

s image by only SIFT at time t+2 (b) Detected feature points in a certain area of right camera s image by

Detected feature points in the right camera s image by proposed Method at time t (c) Detected feature points

certain area of right camera s image by proposed Method at time t+2 Fig.

15 Comparision of the accuracy of the detected feature points Table 2 Accuracy of the detected feature points

14 (a) Detected feature points in the left camera s image by only SIFT at time t (b) Detected feature points in the right camera s image by only SIFT at time t (a) Detected feature points in a certain area of left camera s image by only SIFT at time t+2 (b) Detected feature points in a certain area of right camera s image by only SIFT at time t+2 (c) Detected feature points in the left camera s image by proposed Method at time t (d) Detected feature points in the right camera s image by proposed Method at time t (c) Detected feature points in a certain area of left camera s image by proposed Method at time t+2 (d) Detected feature points in a certain area of right camera s image by proposed Method at time t+2 Fig. 14 Comparision of the number of the detected feature points Fig. 15 Comparision of the accuracy of the detected feature points Table 2 Accuracy of the detected feature points in a certain area Proposed method Only SIFT The number of all feature points 1, The number of accurate feature points 1, Accuracy NURBS 2 3 NURBS NURBS a 17b 2 NURBS 3

Viewpoint from the ground Viewpoint from the side

16 Shape change of the aurora by the time progress

distortion by fish-eye lens (b) The reconstructed

dimensional image Comparison between the shape of

distortion by fish-eye lens and the shape of aurora

15 Viewpoint from the ground Viewpoint from the side Aurora 3D shape by using NURBS t t+ t t+2 t Time Fig. 16 Shape change of the aurora by the time progress Fig. 17 (a) The actual aurora image after eliminating distortion by fish-eye lens (b) The reconstructed aurora image by NURBS reprojected on the two dimensional image Comparison between the shape of actual aurora on the actual image after eliminating distortion by fish-eye lens and the shape of aurora reconstructed by NURBS on the two dimensional projection image SIFT

16 3 2 2 SIFT Aso, T., Ejiri, M., Urashima, A., Miyaoka, H., Steen, A., Brandstorm, U. and Gustavsson, B., First results of auroral tomography from ALIS-Japan multi-station observations in March, 1995, Earth Planets Space, Vol. 50 (1998), pp Benezeth, Y., Jodoin, P. M., Emile, B., Laurent, H. and Rosenberger, C., Review and evaluation of commonlyimplemented background subtraction algorithms, Proceedings of the 17th International Conference on Pattern Recognition (2008), pp Brown, N. B., Davis, N., Hallinan, T. and Stenbeak, N. H., Altitude of pulsating aurora determined by a new instrumental technique, Geophysical Research Letter, Vol. 2, No. 7 (1976), pp Fujii, H., Kubo, T., Yamashita, A., Takeuchi, A., Tanaka, M., Kataoka, R., Miyoshi, Y., Okutomi, M. and Asama, H., Aurora 3D-measurement and visualization using fish-eye stereo camera, Proceedings of ACM SIGGRAPH Asia 2014 Posters, Article Np.24 (2014). Kataoka, R., Miyoshi, Y., Shigematsu, K., Hampton, D., Mori, Y., Kubo, T., Yamashita, A., Tanaka, M., Takahei, T., Nakai, T., Miyahara, H. and Shiokawa, K., Stereoscopic determination of all-sky altitude map of aurora using two ground-based Nikon DSLR cameras, Annales Geophysicae, Vol. 31, No. 9 (2013), pp , (1999),.,,,,,,,, 3, 2014 (2014), pp Lowe, D., Distinctive image features from scale-invariant keypoints, International Journal of Computer Vision, Vol. 60, Issue 2 (2004), pp Mori, Y., Yamashita, A., Tanaka, M., Kataoka, R., Miyoshi, Y., Kaneko, T., Okutomi, M. and Asama, H., Calibration of fish-eye stereo camera for aurora observation, Proceeding of the International Workshop on Advanced Image Technology 2013 (2013), pp ,,,,, Vol. J90-D, No.1 (2007), pp Nishimura, Y., Bortnik, J., Li, W., Thome, R., Lyons, L., Angelopoulos, V., Mende, S., Bonnell, J., Contel, O. L., Cully, C., Ergun, R. and Auster, H. U., Identifying the driver of pulsating aurora, Science, Vol. 330 (2010), pp Sharp, W., Rocket-borne spectroscopic measurements in the ultraviolet aurora: Nitrogen vegard-kaplan bands, Journal of Geophysical Research, ISSN (1971), pp Stenbaek, N. H. and Hallinan, T., Pulsating auroras: Evidence for noncollisional themalization of precipitating electrons, Space Physics, Vol. 84, Issue A7 (1979), pp

17 Störmer, C., Preliminary report on the result of the aurora borealis expedition to bossekop in the Spring of 1913 Third communication, Terrestrial Magnetism and Atmospheric Electricity, Vol. 20, Issue 4 (1915), pp Tanaka, Y., Aso, T., Gustavsson, B., Tanabe, K., Ogawa, Y., Kadokura, A., Miyaoka, H., Sergienko, T., Brandstrom, U. and Sandahl, I., Feasibility study on generalized-aurora computed tomography, Annales Geophysicae, Vol. 29 (2011), pp References Aso, T., Ejiri, M., Urashima, A., Miyaoka, H., Steen, A., Brandstorm, U. and Gustavsson, B., First results of auroral tomography from ALIS-Japan multi-station observations in March, 1995, Earth Planets Space, Vol. 50 (1998), pp Benezeth, Y., Jodoin, P. M., Emile, B., Laurent, H. and Rosenberger, C., Review and evaluation of commonlyimplemented background subtraction algorithms, Proceedings of the 17th International Conference on Pattern Recognition (2008), pp Brown, N. B., Davis, N., Hallinan, T. and Stenbeak, N. H., Altitude of pulsating aurora determined by a new instrumental technique, Geophysical Research Letter, Vol. 2, No. 7 (1976), pp Fujii, H., Kubo, T., Yamashita, A., Takeuchi, A., Tanaka, M., Kataoka, R., Miyoshi, Y., Okutomi, M. and Asama, H., Aurora 3D-measurement and visualization using fish-eye stereo camera, Proceedings of ACM SIGGRAPH Asia 2014 Posters, Article Np.24 (2014). Kataoka, R., Miyoshi, Y., Shigematsu, K., Hampton, D., Mori, Y., Kubo, T., Yamashita, A., Tanaka, M., Takahei, T., Nakai, T., Miyahara, H. and Shiokawa, K., Stereoscopic determination of all-sky altitude map of aurora using two ground-based Nikon DSLR cameras, Annales Geophysicae, Vol. 31, No. 9 (2013), pp Kamide, Y., Aurora A Message from the Sun (1999), YAMA-KEI Publishers Co., Ltd. (in Japanese). Kubo, T., Yamashita, A., Fujii, H., Tanaka, M., Kataoka, R., Miyoshi, Y., Okutomi, M. and Asama, H., 3D measurement of aurora movie photographed by stereo fish-eye camera, Proceedings of the 2014 Japan Society for the Precision Engineering Autumn Meeting (2014), pp (in Japanese). Lowe, D., Distinctive image features from scale-invariant keypoints, International Journal of Computer Vision, Vol. 60, Issue 2 (2004), pp Mori, Y., Yamashita, A., Tanaka, M., Kataoka, R., Miyoshi, Y., Kaneko, T., Okutomi, M. and Asama, H., Calibration of fish-eye stereo camera for aurora observation, Proceeding of the International Workshop on Advanced Image Technology 2013 (2013), pp Nakano, M., Li, S. and Chiba, N., Calibrating fisheye camera by stripe pattern based upon spherical model, Vol. J90-D, No.1 (2007), pp (in Japanese). Nishimura, Y., Bortnik, J., Li, W., Thome, R., Lyons, L., Angelopoulos, V., Mende, S., Bonnell, J., Contel, O. L., Cully, C., Ergun, R. and Auster, H. U., Identifying the driver of pulsating aurora, Science, Vol. 330 (2010), pp Sharp, W., Rocket-borne spectroscopic measurements in the ultraviolet aurora: Nitrogen vegard-kaplan bands, Journal of Geophysical Research, ISSN (1971), pp Stenbaek, N. H. and Hallinan, T., Pulsating auroras: Evidence for noncollisional themalization of precipitating electrons, Space Physics, Vol. 84, Issue A7 (1979), pp Störmer, C., Preliminary report on the result of the aurora borealis expedition to bossekop in the spring of 1913 Third communication, Terrestrial Magnetism and Atmospheric Electricity, Vol. 20, Issue 4 (1915), pp Tanaka, Y., Aso, T., Gustavsson, B., Tanabe, K., Ogawa, Y., Kadokura, A., Miyaoka, H., Sergienko, T., Brandstrom, U. and Sandahl, I., Feasibility study on generalized-aurora computed tomography, Annales Geophysicae, Vol. 29 (2011), pp

xx/xx Vol. Jxx A No. xx 1 Fig. 1 PAL(Panoramic Annular Lens) PAL(Panoramic Annular Lens) PAL (2) PAL PAL 2 PAL 3 2 PAL 1 PAL 3 PAL PAL 2. 1 PAL

xx/xx Vol. Jxx A No. xx 1 Fig. 1 PAL(Panoramic Annular Lens) PAL(Panoramic Annular Lens) PAL (2) PAL PAL 2 PAL 3 2 PAL 1 PAL 3 PAL PAL 2. 1 PAL PAL On the Precision of 3D Measurement by Stereo PAL Images Hiroyuki HASE,HirofumiKAWAI,FrankEKPAR, Masaaki YONEDA,andJien KATO PAL 3 PAL Panoramic Annular Lens 1985 Greguss PAL 1 PAL PAL 2 3 2 PAL DP