(MIRU2009) MCMC-based particle filter, NTT stream processing stream processing 10 stream processing Real time esti

Size: px

Start display at page:

Download "(MIRU2009) MCMC-based particle filter, NTT stream processing stream processing 10 stream processing Real time esti"

たかよしむらかわ
5 years ago
Views:

1 (MIRU2009) MCMC-based particle filter, NTT akisato@ieee.org stream processing stream processing 10 stream processing Real time estimation of human visual attention with MCMC-based particle filter Kouji MIYAZATO,, Akisato KIMURA, Shigeru TAKAGI, Junji YAMATO, and Kunio Abstract KASHINO NTT Communication Science Laboratories, NTT Corporation, Japan. Department of Information and Communication Systems Engineering, Okinawa National College of Technology, Japan. akisato@ieee.org This report proposes a new method for achieving a precise estimation of human visual attention with considerably less execution time. The main contribution of this report is the incorporation of a particle filter with Markov chain Monte-Carlo (MCMC) sampling into a previously proposed stochastic model of saliency-based human visual attention. This enables us to introduce stream processing with such as graphics processing units (GPU) for the acceleration of the estmation. Experimental results indicate that the proposed method can estimate human visual attention more than 10 times faster and more precisely than previous methods. Key words Saliency-based human visual attention, dynamic Bayesian network, stream processing, Markov chain Monte-Carlo (MCMC), particle filter. 1. Introduction [1] [2] [3] [4] Koch Ullman [5] Itti [6] (saliency map) (saliency)

Itti [7], [8] [9], [10] Pang [11] Pang Itti 1 2 1) CPU GPU Cell [12] 2) OpenMP CUDA 1 SDK API stream processing [13] stream processing [14] stream processing Pang stream processing stream processing

2 Itti [7], [8] [9], [10] Pang [11] Pang Itti 1 2 1) CPU GPU Cell [12] 2) OpenMP CUDA 1 SDK API stream processing [13] stream processing [14] stream processing Pang stream processing stream processing Pang stream processing (Markov chain Monte-Carlo or MCMC) 1 home jp.html Fig. 1 1 Our stochastic model of human visual attention MCMC Pang I = I(1 : T ) = {I(t)} T t=1 T I I(t) I t I y I y 2 S = S(1 : T ) = {S(t)} T t=1 S(t) = {s(t, y)} y I s(t, y) t y 3. 3 S = S(1 : T ) = {S(t)} T t=1 S(t) = {s(t, y)} y I s(t, y) t y

3 4. U = u(1 : T ) = {u(t)} T t=1 u(t) 4 X = x(1 : T ) = {x(t)} T t= I S Pang Itti [6] Itti I(t) S(t) i(t) Gaussian pyramid Itti optical flow 9 Gaussian pyramid 6 feature map feature map conspicuity map feature map feature map conspicuity map stream processing Itti GPU [15] CPU 4. S S Pang t S(t) 1 t 1 S(t 1) S(t) p(s(t, y) s(t 1, y)) = N (s(t 1, y), σ s1 ), p(s(t, y) s(t, y)) = N (s(t, y), σ s2 ), σ si (i = 1, 2) N (s, σ) s, σ y s(t, y) s(t) t 1 s(1 : t 1) t 1 s(t 1) p(s(t 1) s(1 : t 1)) = N (ŝ(t 1), σ s (t 1)). t s(t) t s(t) p(s(t) s(1 : t)) = N (ŝ(t), σ s (t)), ŝ(t) = σs2 2 σs1 2 + σ2 s2 + 1) σ2 s(t 1)ŝ(t + σ2 s1 + σ 2 s(t 1) σ 2 s1 + σ2 s2 + σ2 s(t 1) s(t), σ 2 s(t) = σ2 s2 (σ 2 s1 + σ 2 s(t 1)) σ 2 s1 + σ2 s2 + σ2 s(t 1), stream processing stream processing

4 3-5msec/frame CPU Pang stream processing 2 [16] (1) overt shifts of attention (2) covert shifts of attention S(t) u(t) x(t) p(x(t), u(t) p(s(t)), x(t 1), u(t 1)) p(x(t) p(s(t))) p(u(t) u(t 1)) p(x(t) x(t 1), u(t)), (1) p(s(t)) 4. p(s(t)) = {p(s(t, y))} y I, p(s(t, y)) = p(s(t, y) s(1 : t, y)) y I. (1) covert shifts, overt shifts u(t) (1) passive u(t) = 0: (2) active u(t) = 1: t (1) t p(s(1 : t)) x(t) u(t) z(t) p(x(t) p(s(1 : t))) = p(z(t) p(s(1 : t))), (2) p(z(t) p(s(1 : t))) = u(t)=0,1 z(t 1) p(z(t 1) p(s(1 : t 1))) p(z(t) p(s(t)), z(t 1))dz(t 1). (3) p(x(t) p(s(1 : t))) x(t) x(t) X(t) (2)(3) t t 1 (1) (2) N {z n (t) = (x n (t), u n (t))} N n=1 {w n (t)} N n=1 p(z(t) p(s(1 : t))) N w n (t) δ(z(t), z n (t)), (4) n=1 δ(, ) 2 1 (1) 1 p(x(t) p(s(t))) 2 (1) 2 p(u(t) u(t 1)) 3 p(x(t) x(t 1), u(t)) 5. 2 [17] 1 p(x(t) p(s(t))) = p(s(t, x(t)) = s) x =x(t) P (s(t, x) < = s)ds, (5) P (s(t, y) < = s) y p(s(t, y)) Pang (5) p(x(t) p(s(t))) = p(s(t, x(t)) = s) P (s(t, x(t)) < = s) P (s(t, x) < = s)ds. (6) x I

2 2 Fig. 2 Tree-based multiplication with a binary tree and its parallelization (6) s (6) x(t) t s 1 5. 2 2 s (6) x(t) (6) 2 5. 3 5.

3 Strategies for calculating eye focusing density maps (Top) old strategy (Bottom) new strategy 1 (1) p(s(t)) z(t) (1) 2 3 MCMC stream processing 1 t 1

5 2 2 Fig. 2 Tree-based multiplication with a binary tree and its parallelization (6) s (6) x(t) t s s (6) x(t) (6) Pang [11] (1) t 1 {z n (t 1)} N n=1 t {z n (t)} N n=1 [18] stream processing (1) Pang Fig. 3 Strategies for calculating eye focusing density maps (Top) old strategy (Bottom) new strategy 1 (1) p(s(t)) z(t) (1) 2 3 MCMC stream processing 1 t 1 t 1 {z n (t 1) = (x n (t 1), u n (t 1))} N n=1 {w n (t 1)} N n=1 t {z n (t) = (x n (t), u n (t))} N n=1 (1) 2 3 Metropolis [19] u n (t) p(u(t) u n (t 1)), (7) x n (t) p(x(t) x n (t 1), u n (t)). (8) (7) 2 2 (8) Pang

6 p(x(t) x(t 1), u(t)) = L(x(t); x(t 1), γ x,u(t), σ x,u(t) ), γ xi σ xi (i = 0, 1) L(x; x, γ, σ) } ( x x γ)2 L(x; x, γ, σ) exp { 2σ 2. 2 (1) 1 t w n (t) w n (t 1) p(x(t) = x n (t) p(s(t))) (4) {z n (t)} N n=1 {w n (t)} N n=1 3 2 [20] Pang [11] Itti [6] University of South California CRCNS eye MPEG fps fps original experiment [11] normalized scanpath saliency (NSS) [21] Table 1 Platform used in the evaluation OS Windows Vista Ultimate Development Microsoft Visual Studio 2008 C++ platform OpenCV 1.1pre & NVIDIA CUDA 2.1 Optimization none CPU Intel Core2 Quad Q6600 (2.40GHz) RAM 4.0GB GPU NVIDIA GeForce8800GT 2 SLI (512MB) t NSS NSS(t) = 1 N s { N s j=1 1 σ(p(x)) max p(x(t)) p(x) x(t) R j(t) N s R j (t) (j = 1, 2,, N s ) j t 30 p(x; t) t x p(x; t) σ p (x; t) t x Pang 5. 3 MCMC Pang 2 Itti Pang MCMC Itti 6. 2 (3-5 msec/frame) MCMC Pang 1/100 },

2 Itti Itti 7. stream processing MCMC stream processing [22] 8.

7 4 [msec/frame] Fig. 4 Total execution time [msec/frame] 5 [log msec/frame] Fig. 5 Detailed execution time for each step 6 Fig. 6 Average NSS score 2 1/ NSS Pang Itti 2 MCMC Pang Pang Itti NSS 6. 2 Itti Itti 7. stream processing MCMC stream processing [22] 8. the University of South California Laurent Itti Kyungpook National University Minho Lee NTT NTT NTT [1] N. Ouerhani and H. Hügli, Robot self-localization using visual attention, Proc. CIRA, pp , 2005.

図7 出力結果上から入力映像 Itti モデル提案モデル Fig. 7 Snapshots of results (From top: input, Itti s model, and proposed model) [2] T. Xu, Q. Mu hlbauer, S. Sosnowski, K. Ku hnlenz, and M.

8 図7 出力結果上から入力映像 Itti モデル提案モデル Fig. 7 Snapshots of results (From top: input, Itti s model, and proposed model) [2] T. Xu, Q. Mu hlbauer, S. Sosnowski, K. Ku hnlenz, and M. Buss, Looking at the surprise: Bottom-up attetional control of an active camera system, Proc. ICARCV, pp , [3] S. Frintrop, A. Nu chter, H. Surmann, and J. Hertzberg, Saliency-based object recognition in 3D data, Proc. IROS, pp , [4] S. Li and M. Lee, An eﬃcient spatiotemporal attention model and its application to shot matching, IEEE Trans CSVT, Vol.17, No.10, pp , [5] C. Koch and S. Ullman, Shifts in selective visual attention: Towards the underlying neural circuitry, Human Neurobiology, Vol.4, pp , [6] L. Itti, C. Koch, and E. Niebur, A model of saliencybased visual attention for rapid scene analysis, IEEE Trans PAMI, Vol.20, No.11, pp , [7] S. Jeong, S. Ban, and M. Lee, Stereo saliency map considering aﬀective factors and selective motion analysis in a dynamic environment, Neural Networks, Vol.21, pp , [8] D. Gao and N. Vasconcelos, Decision-theoretic saliency: Computational principles, biological plausibility, and implications for neurophysiology and psychophysics, Neural Computation, Vol.21, No.1, pp , [9] L. Itti and P. Baldi, A principled approach to detecting surprising events in video, Proc. CVPR, pp , [10] C. Leung, A. Kimura, T. Takeuchi, J. Yamato, and K. Kashino, A computational model of saliency depletion/recovery phenomena for the salient region extraction of videos, Proc. MIRU, pp , [11] D. Pang, A. Kimura, T. Takeuchi, J. Yamato, and K. Kashino, A stochastic model of selective visual attention with a dynamic Bayesian network, Proc. MIRU, pp , [12] D. Mallinson and M. DeLoura, CELL: A new plat- [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] OS form for digital entertainment, Game Developers Conference, U. Kapasi, S. Rixner, W. Dally, B. Khailany, J.H. Ahn, P. Mattson, and J. Owens, Programmable stream processors, IEEE Computer, Vol.36, No.8, pp.54 62, O. Lozano and K. Otsuka, Real-time visual tracker by stream processing, Journal of Signal Processing Systems, B. Han and B. Zhou, High speed visual saliency computation on GPU, Proc. ICIP, pp , A.R. Hunt and A. Kingstone, Covert and overt voluntary attention: linked or independent?, Cognitive Brain Research, Vol.18, No.1, pp , M.P. Eckstein, J.P. Thomas, J. Palmer, and S.S. Shimozaki, A signal detection model predicts eﬀects of set size on visual search accuracy for feature, conjunction, triple conjunction and disjunction displays, Perception and Psychophysics, Vol.62, pp , C.P. Robert and G. Casella, Monte Carlo Statistical Methods (Springer Texts in Statistics), 2nd ed corr. 2nd printing ed., Springer, N. Metropolis, A. Rosenbluth, M. Rosenbluth, A. Teller, and E. Teller, Equation of state calculations by fast computing machines, Journal of Chemical Physics, Vol.21, pp , B. Ristic, S. Arulampalam, and N. Gordon, Beyond the Kalman ﬁlter: Particle ﬁlters for tracking applications, Artech House Publishers, Boston, R.J. Peters and L. Itti, Beyond bottom-up: Incorporating task-dependent inﬂuences into a computational model of spatial attention, Proc. CVPR, pp.1 8, K. Fukuchi, K. Miyazato, A. Kimura, S. Takagi, and J. Yamato, Saliency-based video segmentation with graph cuts and sequentially updated priors, Proc. ICME, 2009.

258 5) GPS 1 GPS 6) GPS DP 7) 8) 10) GPS GPS 2 3 4 5 2. 2.1 3 1) GPS Global Positioning System

258 5) GPS 1 GPS 6) GPS DP 7) 8) 10) GPS GPS 2 3 4 5 2. 2.1 3 1) GPS Global Positioning System Vol. 52 No. 1 257 268 (Jan. 2011) 1 2, 1 1 measurement. In this paper, a dynamic road map making system is proposed. The proposition system uses probe-cars which has an in-vehicle camera and a GPS receiver.