2/69

Size: px

Start display at page:

Download "2/69"

くうしょうめいこ
5 years ago
Views:

1 SVM (MM ) DC EM l Generative Adversarial Network(GAN) 1/69

2 2/69

3 input x output y = f(x; Θ) Θ : Deep Neural Network(DNN) 3/69

4 f(x; Θ) = ϕ D ( ϕ 2 (b 2 + W 2 ϕ 1 (b 1 + W 1 x)) ), Θ = (b 1, W 1, b 2, W 2,..., b D, W D ). 1. z 0 = x R d 0 2. k = 1,..., D 3. z D z k = ϕ k (W k z k 1 + b k ), W k R d k d k 1, b k R d k, ϕ k : R d k R d k W 2 z 1 + b 2 W 1 x + b 1 z 2 = ϕ 2 (W 2 z 1 + b 2 ) z 0 = x z 1 = ϕ 1 (W 1 x + b 1 ) 4/69

5 ϕ(v) (activation function) ϕ : R R ϕ(v) = (ϕ(v 1 ),..., ϕ(v d )) ϕ : R R 1 Sigmoid function: ϕ(z) = 1 + e z Tangent hyperbolic: ϕ(z) = tanh(z) Rectified linear unit (ReLU): ϕ(z) = max{z, 0} 5/69

6 1000 ( ) 6/69

7 7/69

一般物体検出画像に写っている物体を認識 (多値判別) 一般画像認識のコンペティション (ILSVRC2012

種類の物体を認識するタスク AlexNet GooLeNet ZFNet Ensemble ResNet SENet

com/entry/20151108/1446952402 http://image-net.

8 一般物体検出画像に写っている物体を認識 (多値判別) 一般画像認識のコンペティション (ILSVRC2012 データセット) ILSVRC クラス分類タスクのエラー率 top5 error の推移 120 万枚の画像 1,000 種類の物体を認識するタスク AlexNet GooLeNet ZFNet Ensemble ResNet SENet /69

Convolutional Encoder-Decoder Architectures for Scene

9 セグメンテーションセグメンテーション画像を画素レベルで判別自動運転などに応用 Kendall, et al., Bayesian SegNet: Model Uncertainty in Deep Convolutional Encoder-Decoder Architectures for Scene Understanding, arxiv preprint arxiv: , /69

10 画像スタイル変換 = 畳み込み NN の中間層で抽出される情報画像の位置情報 & スタイル情報位置情報はそのままスタイル情報を他の絵に近づける Gatys, et al., A Neural Algorithm of Artistic Style, CVPR /69

11 DNN 11/69

12 (x 1, y 1 ),..., (x n, y n ) x i R d, y i R 2 l(x, y; Θ) = 1 (y f(x; Θ))2 2 y i R d D 2 l(x, y; Θ) = 1 y f(x; Θ)) /69

13 y i {1, 2,..., G} l {1,..., G} e l R G y i y i {e 1,..., e G }. f(x; Θ) R G l(x, y; Θ) = 1 2 y f(x; Θ) 2, l(x, y; Θ) = log σ y (f(x; Θ)), y {1,..., G} σ y softmax f = (f 1,..., f G ) σ y (f) = e f y G l=1 ef l, y = 1,..., G x y ( ) Pr(y x) σ y (f(x; Θ)) 13/69

14 min Θ L(Θ), L(Θ) = 1 n n i=1 l(x i, y i ; Θ) Θ 1 14/69

15 (Gradient Descent Method; GD) 1. Θ 0 2. t = 0, 1, 2,... ( ) Θ t+1 = Θ t η t Θ L(Θ t ) η t > 0 Θ l(x i, y i ; Θ t ), i = 1,..., n DNN 15/69

16 (Stochastic Gradient Descent Method; SGD) 1. Θ 0 2. t = 0, 1, 2,... ( ) m (x i1, y i1 ),..., (x im, y im ) 1 m Θ t+1 = Θ t η t m Θ l(x i k, y ik ; Θ t ) η t > 0 k=1 SGD: m = 1 mini-batch: 1 < m n. L(Θ) mini-batch E [ 1 m m k=1 Θ l(x i k, y ik ; Θ t ) ] = L(Θ) 16/69

17 mini-batch (mini-batch ) SGD: m = 1 1 (mini-batch ) SGD: 1 < m n 1 DNN mini-batch mini-batch size ( ) m = /69

18 18/69

19 epoch ( ) k epoch k : 1 epoch n mini-batch m SGD = t = n/m 1 epoch 1 epoch 19/69

20 epoch How to Center Deep Boltzmann Machines log-likelihood dd b s ( :0.01) dd b s ( :0.01! 0.001) dd b s ( :0.01! ) 00 ( :0.01) 00 ( :0.01! 0.001) 00 ( :0.01! ) epoch (a) MNIST-Sampled dd b s ( :0.01) dd b s ( :0.01! 0.001) dd b s ( :0.01! ) 00 ( :0.01) 00 ( :0.01! 0.001) 00 ( :0.01! ) epoch (b) MNIST-Threshold igure 18: Evolution of the LL of single trials on the test data of (a) MNIST-Sampled and (b) MNIST-Threshold for dd b s and 00 with 500 hidden units. The models were trained for 1000 epochs with a weight decay of and a momentum of20/69 0.9

21 l(x, y; Θ) = l y (f(x; Θ)) f l(x, y; Θ) = Θ Θ (x; Θ) l y(f(x; Θ)). l y (f) = 1 2 (y f)2 = l y = f y l y (f) = log σ y (f 1,..., f G ) = l y = σ(f) e y, σ(f) = (σ 1 (f),..., σ G (f)) T. note: l y (v 1, v 2,..., v k ) l y = ( l y v 1,..., l ) y T v k f Θ (x; Θ) Θ R m f = (f 1,..., f d ) R d f Θ = ( Θf 1,..., Θ f d ) R m d. 21/69

22 f = (f 1,..., f G ) σ y (f) = e f y G l=1 ef l, y = 1,..., G log σ y (f) log σ y (f) = ( f 1 log σ y (f),..., ) T log σ y (f). f G 22/69

23 f l(x, y; Θ) Θ Θ (back propagation) W k z k 1 + b k = (W k, b k ) ( ) z k 1 1 W k z k 1 v k = W k z k 1 = W k ϕ k 1 (v k 1 ) f(x; Θ) = ϕ D (v D ) = ϕ D (W D ϕ D 1 (v D 1 )) = 23/69

24 f(x; Θ) = ϕ D (v D ) = ϕ D ( ϕ k (W } k ϕ k 1 {{ (v k 1 } )) ) R dd v k ( ) f v k 1 = ϕ k 1 v k 1 W T k f v k R d k 1 d D. ϕ k v k = ( ϕ k,1, ϕ k,2..., ϕ k,dk ) R d k d k v k = W k z k 1 f a W k = f a v k z T k 1 R d k d k 1, a = 1,..., d D 24/69

25 z 0 = x z 1 z D 1 z D = f(x; Θ) v 1 = W 1 z 0 v 2 = W 2 z 1 v D f = ϕ D (v D ) v D v D f = ϕ D 1 (v D 1 )W T f D f, v D 1 v D 1 v D v 1 f a W k = f a v k z T k 1, a = 1,..., d D. W k l(x, y; Θ) = d D a=1 f a W k (x; Θ)( l y (f(x i ; Θ))) a. DNN GPU CPU /69

26 ( ) sin((x 1 + x 2 )x 2 2) 26/69

27 ReLU ϕ(z): sigmoid/tanh z ϕ (z) 0 0 Rectified Linear Unit; ReLU ϕ(z) = max{z, 0} ϕ (0) = 0 ϕ (z) = 1, z > 0, 0, z 0. 27/69

28 SGD 28/69

29 SGD Momentum SGD: { Θ t+1 = Θ t η L } Θ + α(θ t Θ t 1 ) AdaGrad: η t Duchi, et al., Adaptive subgradient methods for online learning and stochastic optimization., JMLR 12.Jul (2011): Adam: (DNN ) Kingma, Diederik, and Jimmy Ba. Adam: A method for stochastic optimization. arxiv preprint arxiv: (2014). 29/69

Optimization: Momentum, AdaGrad/AdaDelta Alec Radford, https://imgur.

gradient): Jumps ahead and recalculate the gradient, check for overshooting.

30 Optimization: Momentum, AdaGrad/AdaDelta Alec Radford, Momentum: Add velocity, like a ball with mass rolling downhill NAG (Nesterov accelerated gradient): Jumps ahead and recalculate the gradient, check for overshooting. AdaGrad/AdaDelta (Duchi, Hazan, Singer, 2011): Keep a history of gradients and decrease learning rate in directions with a history of large gradients (Yue Shi Lai) September 13, / 26 30/69

31 D (DNN) Early stopping Dropout SGD 31/69

32 Srivastava, et al., Dropout: A Simple Way to Prevent Neural Networks from Overfitting, JMLR, 15, pp , SGD p p ( 1/p ) 32/69

33 Srivastava, Hinton, Krizhevsky, Sutskever and Salakhutdinov (a) Standard Neural Net (b) After applying dropout. Figure 1: Dropout Neural Net Model. Left: A standard neural net with 2 hidden layers. Right: An example of a thinned net produced by applying dropout to the network on the left. Crossed units have been dropped. its posterior probability given the training data. This can sometimes be approximated quite well for simple or small models (Xiong et al., 2011; Salakhutdinov and Mnih, 2008), but we would like to approach the performance of the Bayesian gold standard using considerably 33/69

34 L 2 ( ) Baldi and Sadowski, Understanding Dropout, NIPS Bayesian Convolutional Neural Networks ( ) Gal, Ghahramani, Dropout as a Bayesian Approximation, NIPS Kendall, Gal, What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?, NIPS /69

35 (Batch normalization) Ioffe, Szegedy, Batch normalization: accelerating deep network training by reducing internal covariate shift, ICML ϕ(z) = tanh(z) (or sigmoid) z ϕ (z) 0 ( ) 0 x i 0 (normalization): x = (x 1,..., x d ) R d x n(x) = x µ σ (element-wise calc.) µ l = E P [x l ], σ l = V P [x l ], l = 1,..., d 35/69

36 NN x : 1. v k = W k z k 1 2. z k = ϕ k (v k ) normalization: 1. v k = W k z k 1 2. u k = n k (v k ): µ, σ batch normalization. 3. z k = ϕ k (u k ) 36/69

37 normalization SGD (Dropout ) 37/69

38 batch normalization: mini-batch x i1,..., x im mini-batch k v k,i1,..., v k,im mini-batch µ k, σ k (v k R d k ) : bn k (v k ; γ k, β k ) = γ k v k µ k σ k + ε + β k. : γ k, β k : ε 38/69

39 1. v 1 = W 1 x 2. k = 1,..., D 1 : v k+1 = W k+1 ϕ k (bn k (v k ; γ k, β k )) W k, γ k, β k BN v k+1 = W k+1 ϕ k (v k ) 39/69

40 u k = bn k (v k ; γ k, β k ), v k = W k ϕ k 1 (u k 1 ) f(x; Θ) = ϕ D (u D ) = ϕ D ( bn k (W k ϕ k 1 (u k 1 ) }{{} v k ; γ k, β k ) ) = : f u k 1 = f v k = γ k σ k f u k v k f = ϕ k 1 Wk T u k 1 v k u k 1 f v k, ( ) 40/69

41 f a = f a ϕ k 1 (u k 1 ) T, W k v k f = v k µ k f, γ k σ k u k f = f. β k u k a = 1,..., d D 41/69

42 1. Batch normalization DNN f = γ k f, v k σ k u k f = v k µ k f. γ k σ k u k ( ), ( ). 42/69

43 Generative Adversarial Nets (GAN) Goodfellow, et al., Generative Adversarial Nets, NIPS /69

44 (generative models) x 1,..., x n i.i.d. p 0 p 0 (x) ( ) /69

45 x = G(z; Θ g ), z q 0 G(z; Θ g ) DNN q 0 (z) ( ) z q 0 x = G(z ; Θ g ) x = G(z; Θ g ), z q 0 (z) p(x) DNN G(z; Θ g ) ( ) 45/69

46 x p 0 (x) x D(x; Θ d ) [0, 1] D(x; Θ d ) 1/2 x p 0 D x p 0 (x) D(x; Θ d ) x p 0 (x) D(x; Θ d ) (1 D(x; Θ d ) ) max Θ d E x p0 [log D(x; Θ d )] + E z q0 [log (1 D(G(z); Θ d ))] D(x; Θ d ) p 0 (x) p(x) 46/69

47 x p 0 G(z; Θ g ) p D G min Θ g max Θ d E x p0 [log D(x; Θ d )] + E z q0 [log (1 D(G(z; Θ g ); Θ d ))] G(z; Θ g ) 47/69

48 min-max SGD (mini-batch) 1. x 1,..., x m: resampling, z 1,..., z m : q 0 2. Θ d Θ d Θ d + η d δθ d δθ d = 1 m { Θd log D(x m i ) + log(1 D(G(z i ))) } i=1 3. Θ g (z 1,..., z m ) Θ g Θ g η g δθ g δθ g = 1 m Θg log(1 D(G(z i ))) m i=1 48/69

sample, in order to demonstrate that the

49 a) b) c) d) Figure 2: Visualization of samples from the model. Rightmost column shows the nearest training example of the neighboring sample, in order to demonstrate that the model has not memorized the training set. Samples are fair random draws, not cherry-picked. Unlike most other visualizations of deep generative models, these 49/69

50 D(x) max D D (x) = note: D(x) concave {p 0 (x) log D(x) + p(x) log(1 D(x))} dx p 0 (x) p 0 (x) + p(x) ( x ) sign(d (x) 1/2) 2 p(x + 1) = p 0 (x), p(x 1) = p(x), P (y = +1) = P (y = 1) = 1/2 50/69

51 p(x) D (x) p min p:pdf { p 0 (x) log p(x) = p 0 (x) p 0 (x) p 0 (x) + p(x) + p(x) log p(x) } dx p 0 (x) + p(x) { p 0 (x) log p 0 (x) p(x) + p(x) log p 0 (x) + p(x) { p0 (x) + p(x) p 0 (x) = 2 2 p 0 (x) + p(x) log p0 (x) + p(x) 2 ( log 2)dx = 2 log 2 2 } dx p 0 (x) + p(x) p 0 (x) p 0 (x) + p(x) + p(x) p 0 (x) + p(x) log } p(x) dx p 0 (x) + p(x) p(x) = p 0 (x) i.e., G(z; Θ g ), z q(z) p 0 (x) D G DNN 51/69

52 1. max p 0(x) log q + p(x) log(1 q) q (0,1) 2. min r log r + (1 r) log(1 r) r (0,1) 52/69

53 Contrastive divergence, 53/69

54 x = (x 1,..., x d ) {0, 1} d p(x; W, b) = exp{xt W x + x T b}, Z Z W = (w ab ) Sym(d), W aa = 0, b R d b = 0 p(x; W ) 54/69

55 i.i.d. x 1,..., x n, x i = (x i1,..., x id ) {0, 1} d W min W l(w ), l(w ) = 1 n n i=1 log p(x i ; W ) W ab log e xt W x x ext W x }{{} p(x;w ) = l(w ) = 1 n = x a x b x x ax b e xt W x x ext W x x i x T i + E W [xx T ] i 55/69

56 1. η t > 0 W 0 2. t = 0, 1, 2,... W t+1 = W t η t l(w t ) Z E W [xx T ] (2 d ) l(w ) l(w ) : Markov Chain Monte Carlo (MCMC) : Contrastive Divergence 56/69

57 MCMC x = (x 1,..., x d ) x 1 x 2 x d x 1 x ( a) x a 1. for a = 1, 2,..., d, 1, 2,..., d,... x a p(x a x ( a) ; W t ), x a x a. 2. x ( p(x; W ) ) 57/69

58 p(x a x ( a) ; W ) p(x a x ( a) ; W ) = p(x 1,..., x d ; W ) x a p(x 1,..., x d ; W ) = exp{2x a b x bw ab } exp{2 b x bw ab } u U[0, 1] 1, u p(1 x ( a) ; W ) 2. x a = 0, o.w. = exp {2x a(w x) a } exp {2(W x) a } + 1 ( ) 58/69

59 x = ( ) v h, h W = ( W v W vh Wvh T W h ). p(v; W ) h exp{x T W x} exp{v T W v v} h exp{2v T W vh h + h T W h h} 59/69

60 v 1,..., v n (h ) min W l(w ), l(w ) = 1 n n i=1 log h p(v, h; W ) W ab log h ext W x Z = h x ax b e xt W x h ext W x E W [x a x b ] = E W [x a x b v] E W [x a x b ] = l(w ) = 1 n E W [xx T v i ] + E W [xx T ] n i=1 60/69

61 ( ) v i 1 p(h v i ; W ) h (v T i ht ) 2 p(v, h; W ) xx T h p(h v i ; W ) (v, h) p(v, h; W ) ( ) 61/69

62 (Restricted Boltzmann machine; RBM) 2 ( ) ( W v W vh Wvh T W h ) ( = ) O W, p(v, h; W ) exp{v T W h} O W T Hinton and Salakhutdinov, Reducing the Dimensionality of Data with Neural Networks, Science. 313 (5786): , /69

63 v h p(h v) v p(v h) RBM dim v = dim v > dim h input output 63/69

64 Deep Belief Network h 3 Deep Boltzmann Machine h 2 W 3 W h 2 W 2 h 1 W v W 1 v Figure 2: Left: Athree-layerDeepBeliefNetworkandathree-laye astackofmodifiedrbm s,thatarethencomposedtocreateadeep Pr(v, h 1, h 2, h 3 ) = Pr(h 2, h 3 )Pr(h 1 h 2 )Pr(v h 1 ) RBM Consider a two-layer Boltzmann machine (see Fig. 2, right panel) with no within-layer connections. The energy of the Hinton, Osindero, Teh, A fast learning algorithm state {v, h 1, h 2 for deep belief nets. } is defined as: Neural Computation, Vol. 18 Issue 7, pp , E(v, h 1, h 2 ; θ) = v W 1 h 1 h 1 W 2 h 2, (9) note: Deep Boltzmann Machine DBN where θ = {W 1, W 2 } are the model parameters, repre- 64/69

65 RBM p(v h; W ) = exp{vt W h} v exp{vt W h}. v exp{v T W h} = v l exp{v l (W h) l } = l v l exp{v l (W h) l } = l (1 + e (W h) l). 65/69

66 p(v h; W ) = l e v l(w h) l 1 + e (W h) l = l p(v l h; W ). p(v l = 1 h; W ) = e(w h) l 1 + e (W h) l. p(h v; W ) = l e h l(w T v) l 1 + e (W T v) l = l p(h l v; W ). p(h l = 1 v; W ) = e(w T v) l 1 + e (W T v) l. 66/69

67 RBM l(w ) E W [xx T v i ], E W [xx T ] (x = (v T, h T ) T ) h p(h v; W ), v p(v h; W ) Python code: h p(h v; W ) >>> import numpy as np >>> vdim = 10; hdim = 100 # >>> # W >>> W = np.random.normal(size=vdim*hdim).reshape(vdim,hdim) >>> v = np.random.uniform(size=vdim) < 0.2 # v >>> # prob: p(h_l=1 v; W), l=1,..,hdim >>> q = np.exp(np.dot(w.t,v)); prob = q/(1+q) >>> # P(h v; W) h >>> hsample = np.random.uniform(size=hdim) < prob 67/69

68 v i h p(h v i ; W ) h ( ) E W [xx T v i ] (v, h) p(v, h; W ) 1. v 0 = v i ( ) 2. t = 0, 1, 2,..., T h t+1 p(h v t; W ), v t+1 p(v h t+1; W ) 3. (v T, h T ) (v T, h T ) ( ) E W [xx T ] T = 1 contrastive divergence Carreira-Perpiñán and Hinton, On Contrastive Divergence Learning, AISTATS /69

69 MNIST( ) RBM MNIST RBM 28 2 = /69

it-ken_open.key

it-ken_open.key 深層学習技術の進展 ImageNet Classiﬁcation 画像認識音声認識自然言語処理機械翻訳深層学習技術はこれらの分野において特に圧倒的な強みを見せている Figure (Left) Eight ILSVRC-2010 test Deep images and the cited4: from: ``ImageNet Classiﬁcation with Networks et