Overview (Gaussian Process) GPLVM GPDM 2 / 59

daichi@ism.ac.jp 2015-3-3( ) 1 / 59

Overview (Gaussian Process) GPLVM GPDM 2 / 59

(Gaussian Process) y 2 1 0 1 2 3 8 6 4 2 0 2 4 6 8 x x y (regressor) D = { (x (n), y (n) ) } N, n=1 x (n+1) y (n+1), ( ) 3 / 59

(Gaussian Process) y 3 2 1 0 1 2 3 4 8 6 4 2 0 2 4 6 8 x x y (regressor) D = { (x (n), y (n) ) } N, n=1 x (n+1) y (n+1), ( ) 3 / 59

(Gaussian Process) y x x y (regressor) D = { (x (n), y (n) ) } N, n=1 x (n+1) y (n+1), ( ) 4 / 59

y = w 0 + w 1 x 1 + w 2 x 2 + ϵ = (w 0 w 1 w 2 ) 1 +ϵ }{{} w T x 1 = w T x + ϵ x 2 }{{} x ŵ = (X T X) 1 X T y ( ), 5 / 59

(GLM) y = w 0 + w 1 x + w 2 x 2 + w 3 x 3 + ϵ (1) = (w 0 w 1 w 2 w 3 ) 1 +ϵ (2) }{{} w T x x 2 x 3 }{{} = w T ϕ(x) + ϵ ϕ(x) (3) ϕ(x)! 6 / 59

(GLM) (2) 1 0.75 0.5 0.25 0 1 0 1 ϕ(x) = ( (x µ 1) 2 2σ 2, (x µ 2) 2 2σ 2,, (x µ K) 2 ) 2σ 2 (4),! µ = (µ 1, µ 2,, µ K ) 7 / 59

x y R y = f(x) x = (x 1,, x d ) R d y = f(x), x ϕ(x) y = w T ϕ(x) (5) ϕ(x) = (ϕ 1 (x), ϕ 2 (x),, ϕ H (x)) T = (1, x 1,, x d, x 2 1,, x 2 d )T w = (w 0, w 1,, w 2d ) T, y = w T ϕ(x) = w 0 + w 1 x 1 + + w d x d + w d+1 x 2 1 + + w 2d x 2 d. 8 / 59

GP (1) y (1) y (N), y = Φw (Φ : ) ϕ 1 (x (1) ) ϕ H (x (1) ) w 1. = ϕ 1 (x (2) ) ϕ H (x (2) ) w 2... ϕ 1 (x (N) ) ϕ H (x (N) ). y (1) y (2) y (N) w H y Φ w w p(w) = N(0, α 1 I), y = Φw, 0, yy T = (Φw) (Φw) T = Φ ww T Φ T (7) = α 1 ΦΦ T (6) 9 / 59

GP (2) p(y) = N(y 0, α 1 ΦΦ T ) (8), {x n } N n=1 (x 1, x 2,, x N ), y = (y 1, y 2,, y N ), p(y). =, K = α 1 ΦΦ T k(x, x ) = α 1 ϕ(x) T ϕ(x ) (9) k(x, x ) x x ; x y 10 / 59

GP (3), ϵ { y = w T ϕ(x) + ϵ = p(y f) = N(w T ϕ(x), β 1 I) (10) ϵ N(0, β 1 I) f = w T ϕ(x) p(y x) = p(y f)p(f x)df (11) = N(0, C) (12) Gaussian, C : C(x i, x j ) = k(x i, x j ) + β 1 δ(i, j). (13) GP, k(x, x ) α, β. 11 / 59

2.5 3 2 1.5 2 1 0.5 1 y 0 y 0 0.5 1 1 1.5 2 2 2.5 5 4 3 2 1 0 1 2 3 4 5 x Gaussian: exp( (x x ) 2 /l) 3 3 5 4 3 2 1 0 1 2 3 4 5 x Exponential: exp( x x /l) (OU process) 2.5 2 2 1.5 1 1 0.5 y 0 y 0 1 0.5 1 2 1.5 2 3 5 4 3 2 1 0 1 2 3 4 5 x 2.5 5 4 3 2 1 0 1 2 3 4 5 x Periodic: exp( 2 sin 2 ( x x 2 )/l 2 ) Periodic(L): exp( 2 sin 2 ( x x 2 )/(10l) 2 ) 12 / 59

Correlated Gaussian K = 13 / 59

(2) Correlated Gaussian K = 14 / 59

(3) Correlated Gaussian K = 15 / 59

Infinite dimensional Gaussian, (x 1, x 2,, x n ) y = (y 1, y 2,, y n ), y. (x 1, x 2,, x n ), ( ). K K ij = k(x i, x j ) k. 16 / 59

RBF ϕ(x) = exp((x h) 2 /r 2 ) 1, h k(x, x ) = σ 2 H h=1 ϕ h (x)ϕ h (x ) (14) (x h)2 exp ( r 2 ) exp ( (x h) 2 r 2 ) dh (15) = πr 2 exp ( (x x ) 2 ) 2r 2 θ 1 exp ( (x x ) 2 ) θ 2 2 (16) (x, x ) RBF, RBF. θ 1, θ 2 17 / 59

GP y new y Gaussian, p(y new x new, X, y, θ) = p((y, ynew ) (X, x new ), θ) p(y X, θ) [ exp 1 2 ([y, K ynew ] k T k k ] 1 [ ] y y new y T K 1 y) (17) (18) (19) N(k T K 1 y, k k T K 1 k). (20) K = [k(x, x )]. k = (k(x new, x 1 ),, k(x new, x N )). 18 / 59

GP SVR, Ridge, ARD (Cohn+ 2013) ( ) k(x, x ) = σf 2 exp 1 (x k x k )2 2 σk 2 k (21) Model MAE RMSE µ 0.8279 0.9899 SVM 0.6889 0.8201 Linear ARD 0.7063 0.8480 Squared exp. Isotropic 0.6813 0.8146 Squared exp. ARD 0.6680 0.8098 Rational quadratic ARD 0.6773 0.8238 Matern(5,2) 0.6772 0.8124 Neural network 0.6727 0.8103 19 / 59

GP SVR, Ridge, ARD (Cohn+ 2013) ( ) k(x, x ) = σf 2 exp 1 (x k x k )2 2 σk 2 k (22) Model MAE RMSE µ 0.8541 1.0119 Independent SVMs 0.7967 0.9673 EasyAdapt SVM 0.7655 0.9105 Independent 0.7061 0.8534 Pooled 0.7252 0.8754 Pooled &{N} 0.7050 0.8497 Combined 0.6966 0.8448 20 / 59

GP>SVR,, (Cohn+ 2014 etc.)! 21 / 59

GP GP : / X K 1 O(N 3 ) N > 1000, : m X m, X m O(m 2 N) 22 / 59

Subset of Data : K K mm (23), m O(m 3 ), 23 / 59

Subset of Data : K K mm (24), m O(m 3 ), 1.5 1 0.5 0 0.5 1 1.5 15 10 5 0 5. 10. 15.... 24 / 59

(2) Subset of Regressors (Silverman 1985) : m K K nm K 1 mmk mn = K (25) K nm : N m O(m 2 N) 25 / 59

(2) Subset of Regressors (Silverman 1985) : m K K nm K 1 mmk mn = K (26) K nm : N m O(m 2 N), 1.5 1 0.5 0 0.5 1 26 / 59

K, (Quiñonero-Candela & Rasmussen 2005). 27 / 59

(Titsias 2009), Jensen : log p(x)f(x)dx p(x) log f(x)dx X m GP f m, log p(y) = log p(y, f, f m )dfdf m (27) = log q(f, f m ) p(y, f, f m) q(f, f m ) dfdf m (28) q(f, f m ) log p(y, f, f m) q(f, f m ) dfdf m (29), q(f, f m ) 28 / 59

(2) p(y, f, f m ) = p(y f)p(f f m )p(f m ), q(f, f m ) = p(f f m )q(f m ), log p(y) = = = q(f, f m ) log p(y, f, f m) q(f, f m ) dfdf m (30) p(f f m )q(f m ) log p(y f) p(f f m )p(f m ) dfdf m p(f f m )q(f m ) (31) p(f f m )q(f m ) log p(y f)p(f m) dfdf m q(f m ) (32) q(f m )[ p(f f m ) log p(y f)df } {{ } G(f m ) + log p(f ] m) df m q(f m ) (33) 29 / 59

(3) G(f m ), G(f m ) = p(f f m ) log p(y f)df (34) = p(f f m ) ( N2 ) (y log(2πσ2 f)2 ) 2σ 2 df (35) [ = p(f f m ) N 2 log(2πσ2 ) 1 ] 2σ 2 tr(yt y 2y T f +f T f) df = N 2 log(2πσ2 ) 1 [ y T 2σ 2 y 2y T α+α T α+tr ( K nn K nm K 1 ( α = E[f fm ] = K nm Kmmf 1 ) m = log N(y α, σ 2 I) 1 2σ 2 tr ( K nn K nn (36) mmk mn )] (37) ). (38) 30 / 59

(4), log p(y) = = [ q(f m ) q(f m ) G(f m ) + log p(f m) q(f m ) [ log N(y α, σ 2 I) 1 ] df m (39) 2σ 2 tr ( K nn K nn ) + log p(f m) q(f m ) ] df m [ q(f m ) log N(y α, σ2 I) + log p(f m ) q(f m ) Jensen bound, p(x) log f(x) dx log p(x) ] df m (40) 1 2σ 2 tr(k nn K nn) (41) f(x)dx (42) 31 / 59

(5), log N(y α, σ 2 I)p(f m )df m 1 2σ 2 tr(k nn K nn) (K nn = K nm K 1 mmk mn ) (43) α = E[f f m ] = K nm K 1 mmf m, N(y α, σ 2 I)p(f m )df m = N(y 0, σ 2 I + K nn) (44), log p(y) log N(y 0, σ 2 I + K nn) 1 2σ 2 tr(k nn K nn). (45) 32 / 59

(6) log N(y 0, σ 2 I + K nn) 1 2σ 2 tr(k nn K nn) = log N(y 0, σ 2 I + K nn) 1 2σ 2 tr(cov(f f m)) (46) 1 : f m 2 : f m K nn, 1. 33 / 59

GP SVM y = {+1, 1}, p(y f) = σ(y f) (logit) or Ψ(y f) (probit) minimize: log p(y f)p(f X) = 1 N 2 f T K 1 f log p(y i f i ) (47) i=1 SVM Kα = f, w = α i x i w 2 = α T Kα = f T K 1 f, i 1 N minimize: 2 w 2 C (1 y i f i ) + i=1 = 1 N 2 f T K 1 f C (1 y i f i ) +. (48) i=1, SVM hinge loss. 34 / 59

Loss functions Relationships between GPs and Other Models 2 log(1 + exp( z)) log Φ(z) max(1 z, 0) g ǫ(z) 1 0 2 0 1 4. ǫ 0 ǫ z (a) (b) Figure 6.3: (a) A comparison of the hinge error, g λ and g Φ. (b) The ǫ-insensitive error function used in SVR. SVM ME, :, GP classifier ( ) 35 / 59

DP Gaussian process Dirichlet process [ ] GP: (x 1, x 2,, x ), (y 1, y 2,, y ) DP: (X 1, X 2,, X ), Dir(α(X 1 ), α(x 2 ),, α(x )), smoother 36 / 59

Probabilistic PCA (Tipping & Bishop 1999), { yn = Wx n + ϵ ϵ N(0, σ 2 I) (49) L = log p(y n ) = log N(Wx n, σ 2 I) (50) = N 2 ( log 2π + log C + tr(c 1 S) ) (51), C = WW T + σ 2 I (52) S = 1 N YYT. (53) 38 / 59

(2) L = 0, L W W Ŵ U q(λ q σ 2 I) 1 2 (σ 2 = 0 U q Λ 1 2 ) (54) Λ q, U q : YY T q σ 2 = 0 39 / 59

Gaussian Process Latent Variable Models (GPLVM) Probabilistic PCA (Tipping&Bishop 1999): p(y n W, β) = p(y n x n,w, β)p(x n )dx n (55) p(y W, β) = n p(y n W, β) W GPLVM (Lawrence, NIPS 2003): W prior p(w) = D N(w i 0, α 1 I) (56) i=1 p(y X, β) = p(y X, β)p(w)dw (57) ( 1 = (2π) DN/2 exp 1 ) K D/2 2 tr(k 1 YY T ) (58) 40 / 59

GPLVM (2): PPCA Dual log p(y X, β) = DN 2 log(2π) D 2 log K 1 2 tr(k 1 YY T ) (59) K = αxx T + β 1 I (60) X = [x 1,, x N ] T (61) X, L X = αk 1 YY T K 1 X αdk 1 X = 0 (62) X = 1 D YYT K 1 X X U Q LV T (63) U Q (N Q) : YY T Q λ 1 λ Q L = diag(l 1,, l Q ); l i = 1/ λi αd 1 αβ 41 / 59

GPLVM (3) : Kernel log p(y X, β) = DN 2 log(2π) D 2 log K 1 2 tr(k 1 YY T ) K = αxx T + β 1 I, (64) X = [x 1,, x N ] T (65) = K ( k(x n, x m ) = α exp γ 2 (x n x m ) 2) + δ(n, m)β 1 (66) L K = K 1 YY T K 1 DK 1 L = L K x n,j K x n,j Scaled Conjugate Gradient GPLVM in MATLAB: http://www.cs.man.ac.uk/ neill/gplvm/ 42 / 59

GPLVM (4): 0.3 0.2 0.1 0 0.1 0.2 0.3 0.4 0.2 0.15 0.1 0.05 0 0.05 0.1 0.15 0.2 0.25 2.4 2.2 2 1.8 1.6 1.4 1.2 1 0.8 0.6 0.5 0.8 0.6 0.4 0.2 0 0.2 0.4 0.6 0.2 0.1 0 0.1 0.2 0.3 0.4 Figure 1: Visualisation of the Oil data with (a) PCA (a linear GPLVM) and (b) A GPLVM which uses an RBF kernel. Crosses, circles and plus signs represent stratifi ed, annular and homogeneous flows respectively. The greyscales in plot (b) indicate the precision with which the manifold was expressed in data-space for that latent point. The optimised parameters of the kernel were, and f. PPCA( ), GP-LVM( ), Confidence (O(N 3 )): active set ( ), 43 / 59

GPLVM (4): Caveat PCA, Neil Lawrence, 1e-2*randn(N,dims) Scaled conjugate gradient.03.02.01 0.01.02.03 0.03 0.02 0.01 0 0.01 0.02 0.03 44 / 59

GPLVM (5): 1.5 1 0.5 0 0.5 1 1.5 0 20 40 60 80 100 1 0.8 0.6 0.4 0.2 0 0.2 0.4 0.6 0.8 1 1 0.5 0 0.5 1 45 / 59

GPLVM (6): 1.5 1 0.5 0 0.5 1 1.5 1.5 1 0.5 0 0.5 1 1.5 46 / 59

GPLVM (6): 1.5 1.5 1 1 0.5 0.5 0 0 0.5 0.5 1 1 1.5 1.5 1 0.5 0 0.5 1 1.5 1.5 1.5 1 0.5 0 0.5 1 1.5 PCA 47 / 59

GPLVM (7): MCMC 4 1 3 0.5 0 2 1 0 0.5 1 1 2 3 1 0.5 0 0.5 1 Local 4 4 2 0 2 4 6 Global MCMC ( =0.2, 400 iteration) 0 ( GPDM) 48 / 59

GPLVM (8): MCMC (Oil Flow) Local Global MCMC, X, X 49 / 59

Gaussian Process Dynamical Model (Hertzmann 2005) http://www.dgp.toronto.edu/ jmwang/gpdm/ GPLVM, x n x n (GP ). ( ).? 50 / 59

GPDM (2): Formulation (1) { xt = f(x t 1 ; A) + ϵ x,t y t = g(x t ; B) + ϵ y,t, f GP(0, K x ) (67) g GP(0, K y ) (68) p(y, X α, β) = p(y X, β)p(x α). 1 W N ( p(y X, β) = (2π) ND/2 exp 1 ) K Y D/2 2 tr(k 1 Y YW2 Y T ) (69) GPLVM. K Y ( ) RBF 51 / 59

GPDM (3): Formulation (2) 2 Markov N p(x α) = p(x 1 ) p(x t x t 1, A, α) p(a α) da (70) }{{} t=2 Gaussian ( 1 = p(x 1 ) (2π) d(n 1)/2 K X d exp 1 ) 2 tr(k 1 X X X T ) (71) X = [x 2,, x N ] T K X x 1 x N 1 RBF+ ( k(x, x ) = α 1 exp α 2 2 x x 2) + α 3 x T x + α4 1 δ(x, x ). (72) 52 / 59

GPDM (4): Formulation(3) p(y, X, α, β) = p(y X, β)p(x α)p(α)p(β) (73) p(α) i αi 1, p(β) i β 1 i. (74) log p(y, X, α, β) = 1 2 tr(k 1 X X X T ) + 1 2 tr(k 1 Y YW2 Y T ) + d 2 log K X + D 2 log K Y ( ) log W + log α j + log β j j j }{{} ( ) (75). (76) 53 / 59

Gaussian Process Density Sampler (1) 3 3 3 3 2 2 2 2 1 1 1 1 0 0 0 0 1 1 1 1 2 2 2 2 3 3 2 1 0 1 2 3 3 3 2 1 0 1 2 3 3 3 2 1 0 1 2 3 3 3 2 1 0 1 2 3 (a) l x =1, l y =1, α=1 (b) l x =1, l y =1, α=10 (c) l x =0.2, l y =0.2, α=5 (d) l x =0.1, l y =2, α=5 GP prior? p(x) = 1 Φ(f(x))π(x) (77) Z(f) f(x) GP(x) ; π(x) : Φ(x) [0, 1] : ex. Φ(x) = 1/(1 + exp( x)) 54 / 59

Gaussian Process Density Sampler (2) : Rejection sampling p(x) = 1 Φ(f(x))π(x) (78) Z(f) 1. Draw x π(x). 2. Draw r Uniform[0, 1]. 3. If r < Φ(g(x)) then accept x; else reject x Accept N, reject M ( ) Z(f), Φ(g(x)) MCMC! Infinite Mixture 55 / 59

Gaussian process,,,,, (GPLVM, GPDM) 56 / 59

Literature Gaussian Process Dynamical Models. J. Wang, D. Fleet, and A. Hertzmann. NIPS 2005. http://www.dgp.toronto.edu/ jmwang/gpdm/ Gaussian Process Latent Variable Models for Visualization of High Dimensional Data. Neil D. Lawrence, NIPS 2003. The Gaussian Process Density Sampler. Ryan Prescott Adams, Iain Murray and David MacKay. NIPS 2008. Archipelago: Nonparametric Bayesian Semi-Supervised Learning. Ryan Prescott Adams and Zoubin Ghahramani. ICML 2009. 57 / 59

(Pattern Recognition and Machine Learning), Chapter 6. Christopher Bishop, Springer, 2006. http://ibisforest.org/index.php?prml Gaussian Processes for Machine Learning. Rasmussen and Williams, MIT Press, 2006. http://www.gaussianprocess.org/gpml/ Gaussian Processes A Replacement for Supervised Neural Networks?. David MacKay, Lecture notes at NIPS 1997. http://www.inference.phy.cam.ac.uk/mackay/gp/ Videolectures.net: Gaussian Process Basics. http://videolectures.net/gpip06 mackay gpb/ (1)., 2007. http://www.iris.dti.ne.jp/ tmasada/2007071101.pdf 58 / 59

Codes GPML Toolbox (in MATLAB): http://www.gaussianprocess.org/gpml/code/ GPy (in Python): http://sheffieldml.github.io/gpy/ 59 / 59