IPSJ SIG Technical Report Julien Pilet 1 Augmented Reality PC Head Mounted Display Lukas-Kanade Modified Deformable Lukas-Kanade Towards augment

Transcription

1 1 1 1 Julien Pilet 1 Augmented Reality PC Head Mounted Display Lukas-Kanade Modified Deformable Lukas-Kanade Towards augmenting a moving and deforming surface by direct projection Takayuki Nakamura, 1 Yuji Oyamada, 1 Hideo Saito 1 and Julien Pilet 1 In the contet of augmented reality, we aim to project patterns on a moving tetured surface. Such a system can be used to augment a pattern printed on a real object. For eample, it could animate a character on a T-Shirt. Since the surface is moving, it has to be tracked. However, the projection hinder tracking, since it changes the surface appearance. We propose to address this problem by modifying the Lucas-Kanade algorithm, a well known tracking method recently etended to handle deformable surfaces. Our simulated scenario consist of a projector and a camera. Our algorithm iteratively minimizes the difference between the captured image and the target image which combines the original teture and the projection. Preliminary synthetic results demonstrate the efficiency of our approach applied to comple surfaces. 1. PC ( ) () Augmented Reality(AR) AR Computer Vision AR Head Mounted Display 14) 4) 2. AR AR 1 Keio University 1 c 2010 Information Processing Society of Japan

2 2.1 AR AR AR AR Spatial Augmented Reality(SAR) 4) SAR Shader Lamps 2),16) Shader Lamps 3 SAR 3),7),21) 16) ),6) 9),15),20) 19),23) 2.2 AR AR Lucas-Kanade(LK) ( ) 1). 10) 12),17). 14) ( ) 22) 8) 13) 18) LK Deformable Lucas-Kanade SAR 2.2 SAR 3. 2 c 2010 Information Processing Society of Japan

3 (a) (b) T (c) 1 (a) (b) 1 1(a) 1(b) ( ) ( 1(c)) 4. Lucas- Kanade (LK ) 1) LK (Deformable Lucas-Kanade 22) ) 2 LK Modified Deformable Lucas-Kanade (MDLK ) ( 2(a)) ( ) ( 2(b)) 4.1 Lucas-Kanade LK 1) 3 c 2010 Information Processing Society of Japan

4 Hessian 4.2 Modified Deformable Lucas-Kanade MDLK s, p s, p s, p T ()( = (, y)) T s T p T MDLK 2 2D N z 2 s p [ ] T z 2 = s s y p p y [ ] T = s1... sn y s1... y sn p1... pn y p1... y pn (1) i, j, k [1, N] ( i, y i), ( j, y j), ( k, y k ) W (, z 2 ) [ ] i j [ ] T k W (, z 2 ) = b1 b2 b3 (2) y i y j y k (b 1, b 2, b 3 ) LK Hessian MDLK LK Hessian MDLK z 2 z 2 I() W z 2 I(W (, z 2 )) appearance E a E a(z 2) = [T s(w (, z 2)) + T p(w (, z 2)) I(W (, z 2))] 2 (3) 3 2N 2 regularization E r E r (z 2 ) = z T 2 K z 2 (4) K R 4N 4N 2N 2N K [ ] [ ] K 0 K T K 0 K =, K = 0 K 0 K T K K (i, j, k) i, j, k 1, 2, 1 0 sparse and banded matri 2 E A λ r (5) E A (z 2 ) = E a (z 2 ) + λ r E r (z 2 ) (6) 6 [T s (W (, z 20 )) + T s z 2 + T p (W (, z 20 )) + T p z 2 I(W (, z 2 ))] 2 z 2 z 2 +λ r (z 2 + z 2 ) T K (z 2 + z 2 ) (7) T s, T p W (, z 20 ) z 20 3 W (, z 20 ) T (W (, z 20 )) = T () z [ T s + T p ] T [T s () + T s z 2 + T p () + T p z 2 z 2 z 2 z 2 z 2 I(W (, z 2 ))] 2 + λ r K (z 2 + z 2 ) (8) 7 z z 2 4 c 2010 Information Processing Society of Japan

5 3 ( 4(b)) ( 4(c)) : [pi] : [pi] : [pi] CPU : Intel Core2 Duo CPU 2.80GHz : 3.5GB λ r = z 2 = H 1 4 [ T s z 2 + T p z 2 ] T [I(W (, z 2 )) T s () T p ()] λ r H 1 4 K z 2 (9) H 4 4N 4N Hessian H 4 = [ T s + T p ] T [ T s + T p ] + λ r K (10) z 2 z 2 z 2 z 2 HessianH 4 z 2 z 2 z 2 MDLK z 2 z 2 z 2 (11) ( 1 ) T s T p T ( 2 ) z 20 ( 3 ) I z 2 ( 4 ) 7 z 2 ( 5 ) z 2 11 ( 6 ) 4,5 5. ( 4(a)) (a) (b) (c) pi 1/3 10pi z 2 z 20 ( 1 ) ( ): 0 10pi ( 2 ) ( ): pi ( 3 ) ( ): 0 5pi 0 10pi 5 c 2010 Information Processing Society of Japan

6 5 T I(W (, z 2)) Root Mean Square Error(RMSE) iteration RMSE RMSE 5 RMSE RMSE RMSE RMSE 1 RMSE (pi) (pi) fps 12.7fps 12.7fps 12fps 5 6 I(W (, z 2)) MDLK 6. LK DLK Modified Deformable Lucas-Kanade (MDLK ) DLK Hessian GPU 1) SIMON BAKER and IAIN MATTHEWS. Lucas-kanade 20 years on: A unifying framework. International Journal of Computer Vision 56(3), , 2004, pp , ) Deepak Bandyopadhyay, Ramesh Raskar, and Henry Fuchs. Dynamic shader lamps : Painting on movable objects. In IEEE and ACM International Symposium on Augmented Reality, pp , ) François Bérard. The magic table: Computer-vision based augmentation of a whiteboard for creative meetings. In International Workshop on Projector-Camera Systems (PROCAMS), ) Oliver Bimber and Ramesh Raskar. Spatial Augmented Reality: Merging Real and Virtual Worlds. A K Peters LTD, ) Daniel Cotting, Martin Naef, Markus Gross, and Henry Fuchs. Embedding imperceptible patterns into projected images for simultaneous acquisition and display. In IEEE/ACM International Symposium on Mied and Augmented Reality (ISMAR), pp , ) Daniel Cotting, Remo Ziegler, Markus Gross, and Henry Fuchs. Adaptive instant displays: Continuously calibrated projections using per-piel light control. Computer Graphics Forum, Vol.24, No.3, pp , ) Shilpi Gupta and Christopher Jaynes. The universal media book: Tracking and augmenting moving surfaces with projected information. In International Sympo- 6 c 2010 Information Processing Society of Japan

7 sium on Mied and Augmented Reality (ISMAR), pp , ) A. Hilsmann and P. Eisert. Tracking and Reteturing Cloth for Real-Time Virtual Clothing Applications. Computer Vision/Computer Graphics Collaboration Techniques, p.94, ) Tyler Johnson and Henry Fuchs. Real-time projector tracking on comple geometry using ordinary imagery. In IEEE International Workshop on Projector-Camera Systems (PROCAMS), ) V.Lepetit and P.Fua. Monocular model-based 3d tracking of rigid objects. Now Publishers Inc, ) V.Lepetit, P.Lagger, and P.Fua. Randomized trees for real-time keypoint recognition. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2005, Vol.2, ) D.G. Lowe. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, Vol.60, No.2, pp , ) Francesc Moreno-Noguer, Mathieu Salzmann, Vincent Lepetit, and Pascal Fua. Capturing 3d stretchable surfaces from single images in closed form. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), ) J.Pilet, V.Lepetit, and P.Fua. Fast non-rigid surface detection, registration and realistic augmentation. International Journal of Computer Vision, Vol.76, No.2, pp , ) Ramesh Raskar, Jeroen van Baar, Paul Beardsley, Thomas Willwacher, Srinivas Rao, and Clifton Forlines. ilamps: geometrically aware and self-configuring projectors. ACM Transactions on Graphics, Vol.22, pp , ) Ramesh Raskar, Greg Welch, Kok lim Low, and Deepak Bandyopadhyay. Shader lamps: Animating real objects with image-based illumination. In Eurographics Workshop on Rendering, pp , ) E.Rosten and T.Drummond. Fusing points and lines for high performance tracking. In Tenth IEEE International Conference on Computer Vision, ICCV 2005, Vol.2, ) M.Salzmann, R.Urtasun, and P.Fua. Local deformation models for monocular 3d shape recovery. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1 8, June ) Michael Waschbüsch, Stephan Würmlin, Daniel Cotting, Filip Sadlo, and Markus Gross. Scalable 3d video of dynamic scenes. The Visual Computer, Vol. 21, pp , ) Ruigang Yang and Greg Welch. Automatic and continuous projector display surface calibration using every-day imagery. In International Conference in Central Europe on Computer Graphics, Visualization and Computer Vision, ) Tatsuo Yotsukura, Shigeo Morishima, Frank Nielsen, Kim Binsted, and ClaudioS. Pinhanez. Hypermask - projecting a talking head onto a real object. The Visual Computer, Vol.18, No.2, pp , April ) Jianke Zhu, Michael R. Lyu, and Thomas S. Huang. A fast 2d shape recovery approach by fusing features and appearance. Pattern Analysis and Machine Intelligence, IEEE Transactions on, Vol.31, No.7, pp , July ) Stefanie Zollmann, Tobias Langlotz, and Oliver Bimber. Passive-active geometric calibration for view-dependent projections onto arbitrary surfaces. Journal of Virtual Reality and Broadcasting, Vol.4, No.6, pp. 1 11, c 2010 Information Processing Society of Japan

8 (a) I(W (, z 2)) (b) (c) I(W (, z 2 )) (d) 6 8 c 2010 Information Processing Society of Japan