Vol. 44 No. SIG 9(CVIM 7) July 2003, Robby T. Tan, 1 Estimating Illumination Position, Color and Surface Reflectance Properties from a Single Image Kenji Hara,, Robby T. Tan, Ko Nishino, Atsushi Nakazawa, and Katsushi Ikeuchi In this paper we propose a new method for estimating a position and color of a light source, as well as reflectance properties of a real object s surface, from a single image. We use the intensity of the diffuse and specular component for estimating the light source position, while the color distribution of the specular region is for estimating the light source color. The flow of this method is basically as follows: first, an initial position of the light source is estimated from a peak location of the specular region and a rough intensity value of the diffuse region. This diffuse-to-specular intensity value is also used to determine the initial values of the object reflectance properties. After having the initial values, using iterative fitting method, the light position and reflectance properties are estimated simultaneously. Finally, the estimation process of the light source color is based on the color distribution of the specular region. By knowing the light source position, color and the object reflectance properties, we can freely generate synthetic images under arbitrary light source conditions. 1. 1 Institute of Industrial Science, The University of Tokyo Department of Computer Science, Columbia University Presently with Fukuoka Industrial Technology Center Presently with Cybermedia Center, Osaka University 2 1 1),2) Ikeuchi Torrance-Sparrow 3) 4) Ramamoorthi 5) Tominaga Phong 1 6) 94
Vol. 44 No. SIG 9(CVIM 7) 95 1 1) 2) 1) 1 2) 3 7) 1) 2) 3 3) 4) 5) 2 3 4 5 2. 2 2 (a) (d) (g) (b) (e) (h) No Convergence? End (f) (c) Yes * ** * ** 1 Fig. 1 Outline of the overall algorithm. 1) 2) 1) 2 2.1 1 (1) 1 (a) 1 (b) (2) 1 (c) (3) (2)
96 July 2003 1 (d) (4) 1 (e) 1(f) (5) 1 (g) r-g 1 (h) 2.2 Torrance-Sparrow 3) [ ] i c = k d,c cos θ i + k s,c exp[ α2 cos θ r 2σ ] Lc 2 R (1) 2 4) c RGB i c L c R 2 k d,c k s,c σ θ i θ r α 2 2 (1) k d,c k s,c L c K d,c = k d,c L c (2) K s,c = k s,c L c (3) (1) I c = K d,c cos θ i + K s,c cos θ r exp[ α2 2σ 2 ] (4) I c = i c R 2 (5) (4) K d = [K d,r,k d,g,k d,b ] T K s =[K s,r,k s,g,k s,b ] T σ K d K s σ (2) (3) K d K s incident light surface normal bisector object surface 2 Fig. 2 Geometric model. view RGB RGB c RGB 2.2.1 N p L p V p L p N p V p L p = N p +(N p, V p )N p V p (6) N p V p 3 2 L p L 3 P L p t L = P + tl p (7) L t (4) 1 t f N j ( f(x 1 (t),,x Nj (t))= x j (t) 1 N j ) 2 x l (t) N j j=1 l=1 (8) N j x j (t)
Vol. 44 No. SIG 9(CVIM 7) 97 I (j) x j (t) = (9) cos(θ (j) i (t)) I (j) θ (j) i (t) j f (8) t f t t 1 = t t t 2 = t t 3 = t+ t 3 f(x 1 (t n ),,x Nj (t n )) (n =1, 2, 3) f t n t t =2mm t (7) L t K d 2 N j ( 2 E 1 (K d )= I (j) K d cos(θ (j) i (t ))) (10) j=1 K d N j / Nj Kd = I (j) cos(θ (j) i (t )) (11) j=1 j=1 2.2.2 N k k I (k) s = I (k) K d cos(θ (k) i (t )) (12) (4) 2 I s = K ] s exp [ α2 (13) cos θ r 2σ 2 Y = 1 σ 2 X +lnk s (14) 4) X = α2 2 (15) Y =lni s + ln cos θ r (16) N k (15) (16) X-Y (14) 2 (14) K s + σ + K s + σ + 2 E 2 (K s,σ) N k ( = I s (k) K s [ cos(θ (k) ) exp (α(k) (t )) 2 ] ) 2 2σ 2 k=1 N k K s = k=1 I (k) s r (17) (17) 1) 2) 1) σ K s / Nk [ α(k) (t ) 2 k=1 1 cos(θ r (k) ) exp 2) K s 2σ 2 ] (18) κ (n) = 1 (19) σ (n) N k ( κ (n+1) = κ (n) γ k=1 I (k) s K s cos(θ (k) r ) ] exp [ ) (α(k) (t )) 2 (κ (n) ) 2 2 Ks (α (k) (t )) 2 κ (n) 2 cos(θ (k) r ) ] exp [ (α(k) (t )) 2 (κ (n) ) 2 (20) 2 σ (n+1) = 1 (21) κ (n+1) 3 σ n γ γ γ =1.0 10 7 K s σ 8)
98 July 2003 N k I (k) d = I (k) K s [ cos(θ (k) ) exp (α(k) (t )) 2 ] 2(σ ) 2 r (22) 2 (17) (14) 2.3 Torrance-Sparrow 2 I c = w B (θ) S(λ)E(λ)R c (λ)dλ + w I (θ) E(λ)R c (λ)dλ (23) 2.2 c RGB I c RGB 1 2 λ [400, 700] nm w B (θ) w I (θ) R c (λ) S(λ) E(λ) (23) RGB (R, G, B)=w B (θ)(r, G, B) B +w I (θ)(r, G, B) I (24) r = R R + G + B, g = G R + G + B (25) (r, g) =w B (θ)(r, g) B + w I (θ)(r, g) I (26) RGB (25) r-g RGB Ψ c R c (λ) Ψ c = M(λ, T )R c (λ) dλ (27) M(λ) T Kelvin c 1 M(λ, T )= λ 5[ exp( c 2 λt ) 1] (28) c 1 =3.7418 10 16 Wm 2 c 2 =1.4388 10 2 mk r-g (26) (28) Planckian locus (1) T [T min,t max ] (28) T min =1, 000 T max =10, 000 =1 (2) rg (1) M(λ, T )(T min T T max ) RGB (27) rg (3) (2) rg r-g (26) (r,g ) T color constancy Finlayson 7) (26) 3 (a) Finlayson 3 (b) r-g r g Planckian locus (r, g) g =1 r
Vol. 44 No. SIG 9(CVIM 7) 99 Fig. 3 (a) (b) 3 (a) (b) Method for estimating illumination color: (a) conventional method, (b) proposed method. 4,700 Kelvin 3 (a) 3 (b) 1% 3. 3.1 ( K d, K s,σ) (θ i,θ r,α) R (4) 3.2 E(λ) T M(λ, T ) (23) I c = w B (θ) + w I (θ) S(λ)M(λ, T )R c (λ) dλ M(λ, T )R c (λ) dλ (29) M(λ, ) (28) (29) W = M(λ, T new )R c (λ) dλ M(λ, T )R c (λ) dλ WI c = w B (θ) +w I (θ) S(λ)M(λ, T )R c (λ) dλ M(λ, T )R c (λ) dλ (30) M(λ, T new )R c (λ) dλ M(λ, T new )R c (λ) dλ (31) T new I new c I new c = w B (θ) S(λ)M(λ, T new )R c (λ) dλ + w I (θ) M(λ, T new )R c (λ) dλ (32) (31) (32) R c (λ) δ(λ λ k ) 1 S(λ k )M(λ k T new )
100 July 2003 Table 1 1 Results of estimation. [mm] ( 400.42, 488.34, 797.65) ( 408.88, 493.59, 804.83) [Kelvin] 4,800 4,700 (K s,r,k s,g,k s,b ) (0.123, 0.0258, 0.249) (K d,r,k s,g,k d,b ) (0.494, 0.840, 0.576) σ 0.0702 WI c Ic new (33) δ( ) (1) 2.3 (3) T M(λ, T ) (28) (2) T new M(λ, T new ) (28) r-g 3.2 (2) (r new,g new ) T new (3) W (1) (2) (30) W (4) (33) I c (c = R, G, B) W 4. 4 (a) SONY DXC-9000 4 (b) 4 (c) 5 (a) 5 (b) 5 (c) 1 4 3 2 4 6 7 6 (a) 6 (b) 6 (a) 6 (c) 5,860 Kelvin 7 (a) 7 (b) 7 (c) 8 9 8 20 60 100
Vol. 44 No. SIG 9(CVIM 7) 101 32 0 Fig. 4 4 (a) (b) (c) Synthetic image: (a) real image, (b) virtual object image, (c) error map. 32 0 Fig. 5 5 (a) (b) (c) Synthetic image: (a) real image, (b) virtual object image, (c) error map. 32 0 6 (a) (b) (c) Fig. 6 Synthetic image: (a) real image, (b) virtual object image under new illumination position, (c) error map. 32 0 7 (a) (b) (c) Fig. 7 Synthetic image: (a) real image, (b) virtual object image under new illumination color, (c) error map. 9 8 z cm
102 July 2003 10 (a) 10 (b) 10 (c) 10 (a) 11 (a) 11 (b) 8 9 Fig. 8 Robustness analysis (surface reflection property). Fig. 9 Robustness analysis (illumination position). 10 (a) (b) (c) Fig. 10 Synthetic image: (a) real image, (b) virtual object image under the estimated illumination position and reflection parameters, (c) virtual object image under new illumination position. 11 (a) (b) (c) Fig. 11 Synthetic image: (a) white-balanced image, (b) virtual object image under the estimated illumination color, (c) virtual object image under new illumination color.
Vol. 44 No. SIG 9(CVIM 7) 103 11 (c) 5. 1 Lambertian Torrance-Sparrow NEDO 1) Boivin, S. and Gagalowicz, A.: Image-based rendering of diffuse, specular and glossy surfaces from a single image, Computer Graphics Proceedings, SIGGRAPH2001, pp.107 116 (2001). 2) 3 Vol.41, No.SIG 10(CVIM 1), pp.1 11 (2000). 3) Torrance, K.E. and Sparrow, E.M.: Theory of off-specular reflection from roughened surfaces, Journal of the Optical Society of America, Vol.57, pp.1105 1114 (1967). 4) Ikeuchi, K. and Sato, K.: Determining reflectance properties of an object using range and brightness images, IEEE Trans. Pattern Analysis and Machine Intelligence, Vol.13, No.11, pp.1139 1153 (1991). 5) Ramamoorthi, R. and Hanrahan, P.: A signal processing framework for inverse rendering, Computer Graphics Proceedings, SIG- GRAPH2001, pp.117 128 (2001). 6) Tominaga, S. and Tanaka, N.: Estimating reflection parameters from a single color image, IEEE Computer Graphics and Applications, Vol.20, No.5, pp.58 66 (2000). 7) Finlayson, G.D. and Schaefer, G.: Solving for color constancy using constrained dichromatic reflection model, Inter. J. Computer Vision, Vol.42, No.3, pp.127 144 (2001). 8) Chan, T.F. and Wong, C.K.: Convergence of the alternating minimization algorithm for blind deconvolution, Linear Algebra and its Applications, Vol.316, 1-3, Sep. 2000, pp.259 285 (2000). 9) http://radsite.lbl.gov/radiance/home.html ( 14 9 6 ) ( 15 3 28 ) 1988 1990 1992 1999 2001 4 IEEE Robby T. Tan 2001
104 July 2003 1997 1999 2002 2002 physics-based vision image-based rendering 1999 VSMM 2000 IEEE 1974 1997 1999 2001 2003 3 IEEE 1973 1978 MIT CMU 1996 ICCV-90 CVPR-91 AIJ-92-97 IEEE R&A -98 MIRU2000 11 OSA IEEE Fellow