Convolutional Neural Network A Graduation Thesis of College of Engineering, Chubu University Investigation of feature extraction by Convolution

Size: px

Start display at page:

Download "Convolutional Neural Network A Graduation Thesis of College of Engineering, Chubu University Investigation of feature extraction by Convolution"

たかよしたつざわ
5 years ago
Views:

1 Convolutional Neural Network A Graduation Thesis of College of Engineering, Chubu University Investigation of feature extraction by Convolutional Neural Network Fukui Hiroshi

2 [1] ( ) Deep Learning Deep Learning [2] Drop out[3] Deep Learning Histgram of Oriented Gradients(HOG) [4] Scale Invariant Feature Transform(SIFT) [5] Deep Learning Convolutional Neural Network(CNN) [6] CNN 1989 ii

3 CNN 5 CNN Deep Learning CNN CNN CNN iii

4 Deep Learning Convolutional Neural Network CNN iv

5 40 41 v

6 CNN [7][8] CNN CNN MNIST Dataset vi

7 3.12 CNN epoch vii

8 viii

9 () w 1

10 1.1. f 1.1: x i w i f y x w d w i y (1.1) ( d ) y = f w i x i i=1 (1.1) y f x i w i X (1.2) X 2 (1.2) X 0 1 X 0-1 2

11 (X > 0) f (X) = 1 (X 0) (1.2) 1.2 (1.3) : f (X) = exp ( gx) (1.3) (1.3) g g

12 f(x) (1.4) f (X) = gf (X) (1 f (X)) (1.4) [9] s 1 θ s 2 s 3 a 1 a 2 w j s i w ij a j s d a J S A O 1.3: d J 1 w ij w j θ θ 4

13 1.3. A = [a 1, a 2,..., a j,..., a J ] T w = [w 1, w 2,..., w j ] O (1.5) J O = f w j1 a j θ (1.5) j=1 O T (1.6) (1.7) w t+1 = w t + η (T O) A (1.6) θ t+1 = θ t + η (T O) (1.7) (1.6) (1.7) t η η > 0 (1.6) (1.7) 0% x 1 x (a) 2 x 1 x 2 1.4(b) c d J 5

14 1.3. x 2 x 2 x 1 x 1 1.4: (1.9) E N = 1 N c (T k O k ) 2 (1.8) 2 n=1 k=1 6

15 1.3. θ γ s 1 s 3 a 1 o 1 o 2 s i w ij a j w ij o k a J s d o c S A O 1.5: E N (1.9) E N η w t+1 = w t η E N w t (1.9)

16 (1.10) E n = 1 c (T k O k ) 2 (1.10) 2 k=1 (1.10) E n E n η (1.11) w t+1 = w t η E n w t (1.11) mini batch mini batch mini batch 1 mini batch M (1.12) E m (1.13) E M = 1 M c (T k O k ) 2 (1.12) 2 m=1 k=1 8

17 1.3. w t+1 = w t η E M w t (1.13) θ γ i j k k d c c S A O T 1.6: d c S i A j O k T k f w ij w jk j (1.14) w ij S j θ j 9

18 1.3. d A j = f w ij S i + θ j (1.14) i= (1.10) (1.10) O k O k (1.15) T δ k E n O k = (O k T k ) = δ k (1.15) O k P k = j w jk A j + γ k O k (1.16) O k = f (P k ) (1.16) (1.16) k > 2 (1.17) (1.17) P k 1 P j O k = exp (P k) j exp (P j ) (1.17) E n E n E n (1.18) E n = E njk w jk (1.18) 10

19 1.3. E njk E njk (1.18), (1.19) E njk = E n w jk = E n O k O k w jk = E n O k O k P k P k w jk (1.19) (1.19) E njk (1.20) E njk = E n w jk = δ k O k (1 O k ) A j (1.20) E nij E nij (1.21) E nij = E n w ij = E n A j A j P j P j W ij = E n O k P k A j P j O k P k A j P j w ij ( = δ k O k (1 O k ) W jk ) A j (1 A j ) S i (1.21) k (1.20) (1.21) (1.20) (1.20) (1.11) (1.22) (1.23) w t+1 jk = w t jk η δ k O k (1 O k ) A j (1.22) γ t+1 j = γ t j η δ k O k (1 O k ) (1.23) 11

20 1.3. (1.24) (1.25) ( wij t+1 = wij t η δ k O k (1 O k ) W jk ) A j (1 A j ) S i (1.24) θ t+1 j = θ t j η k ( k δ k O k (1 O k ) W jk ) A j (1 A j ) (1.25) epoch 1epoch t t = 1 C 1 t = 0 C 2 y (1.3) y(x, w) p(c 1 x) p(c 2 x) 1 y(x, w) (1.26) p (t x, w) = y (x, w) t {1 y (x, w)} 1 t (1.26) (1.26) (1.27) N E = {t n ln y n + (1 t n ) ln (1 y n )} (1.27) n=1 12

21 1.3. Simard [10] (1.28) N C E = t nc ln y c (1.28) n=1 c=1 13

22 2 Deep Learning Deep Learning 2.1 O T O k i j j j k k d c c S A O T 2.1:

23 2.1. Convolutional Neural Network Convolutional Neural Network(CNN) CNN CNN 2.1 Convolutional Neural Network CNN CNN Hubel Wisel [11] Hubel 2.2(a) Fukushima 2.2(b) Neocognitron [8] Neocognitron CNN Neocognitron 2.2: CNN [7][8] CNN CNN 15

24 2.1. Convolutional Neural Network 2.3 CNN 2.3: CNN CNN 16

25 2.1. Convolutional Neural Network CNN Hubel n x n y n w n w n x, n y (2.1) n x = n x n w + 1 n y = n y n w + 1 (2.1) Hubel CNN 2.4 P h i h (2.2) i P h = max i P h i (2.2) 2 2 (2.3) n x = n x /2 n y = n y /2 (2.3) 17

26 2.1. Convolutional Neural Network P P 1 P 2 P 3 P 4 2.4: CNN 2.5 Full-Connect Full-Connect 18

27 2.1. Convolutional Neural Network 2.5: CNN (2.1) (2.3) n w n w n w n w Full-Connect 2.6 CNN 19

28 2.1. Convolutional Neural Network 2.6: CNN 20

29 3 CNN CNN CNN 2 2 MNIST Dataset CNN 3.1 MNIST Dataset 3.1 MNIST Dataset MNIST Dataset ,000 10,000 10, pixel 21

30 : MNIST Dataset 22

31 CNN Full-Connect CNN epoch CNN CNN

32 : 24

33 epoch 2 epoch 3.3: 3.4(a) 3.4(b) 3.4(a) 3.4(b) 25

34 Convolutional Neural Network Multi Layer Perceptron Cross entropy e epoch 100 Convolutional Neural Network Multi Layer Perceptron Miss rate[%] epoch :

35 3.3. epoch : 3.4(a) epoch

36 3.4. CNN 100 Convolutional Neural Network Multi Layer Perceptron 1000 Convolutional Neural Network Multi Layer Perceptron Miss rate[%] 10 Cross entropy epoch 1e epoch 3.6: 3.4 CNN CNN CNN CNN 28

37 3.4. CNN CNN CNN : CNN

38 3.4. CNN 3.1: (a) 3.8(c) CNN (MLP) 3.8(a) 3.8(c) CNN (a) 3.8(c) 10%

39 3.4. CNN [%] MLP CNN [pixel] [%] 15 MLP CNN [ ] [%] MLP CNN : 31

40 3.4. CNN 3.9: 32

41 3.4. CNN CNN CNN CNN 3.10: 3.11 CNN CNN 5pixel 33

42 3.4. CNN CNN : 3.13(a) CNN 3.13(b) 3.13(b) 3.13(a) 3.13(b) CNN CNN 34

43 3.4. CNN 3.2 CNN CNN 3.2 CNN 3.2: 2pixel 2pixel 20 MLP CNN CNN CNN epoch 3.13 epoch 3.13 CNN CNN epoch 3.2 CNN epoch 35

44 3.4. CNN Parallel Shift Learning CNN Rotation Learning CNN No Random Learning CNN Cross entropy e-005 1e epoch 100 Parallel Shift Learning CNN Rotation Learning CNN No Random Learning CNN 10 Miss rate[%] epoch : CNN

45 3.4. CNN 3.13: epoch 37

46 Convolutional Neural Network 1 2 Deep Learning Convolutional Neural Network Convolutional Neural Network Deep Learning Convolutional Neural Network Convolutional Neural Network 3 Convolutional Neural Network Convolutional Neural Network Convolutional Neural Network Convolutional Neural Network 38

47 Convolutional Neural Network 39

48 40

49 [1] D. Rumelhart, G. Hintont, and R. Williams, Learning representations by backpropagating errors, Nature, pp , [2] G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, Improving neural networks by preventing co-adaptation of feature detectors, CoRR, vol.abs/ , [3] A. Krizhevsky, I. Sutskever, and G. E. Hinton, Imagenet classification with deep convolutional neural networks, Advances in Neural Information Processing Systems 25, pp , [4] N. Dalal, and B. Triggs, Histograms of oriented gradients for human detection, International Conference on Computer Vision & Pattern Recognition, vol.2, pp , [5] D. G. Lowe, Object recognition from local scale-invariant features, Proceedings of the International Conference on Computer Vision-Volume 2 - Volume 2, pp.1150, IEEE Computer Society, [6] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel, Backpropagation applied to handwritten zip code recognition, Neural Computation, vol.1, pp , [7] J. W. Kimball, Kimball s biology pages,, 2000, biology2/ultranet/visualprocessing.html. 41

50 [8] K. Fukushima, and S. Miyake, Neocognitron: A new algorithm for pattern recognition tolerant of deformations and shifts in position, Pattern Recognition, vol.15, no.6, pp , [9] F. Rosenblatt, The perceptron: A probabilistic model for information storage and organization in the brain, Psychological Review, vol.65, no.6, pp , [10] S. Patrice, V. Bernard, L. Yann, and D. John S, Tangent Prop: a formalism for specifying selected invariances in adaptive networks, NIPS, pp , [11] B. Y. D. H. Hubel, and A. D. T. N. Wiesel, Receptive fields, binocular interaction and functional architecture in the cats visual cortex, The Journal of physiology, vol.160, pp ,

51 Convolutional Neural Network ()

untitled

untitled c ILSVRC LeNet 1. 1 convolutional neural network 1980 Fukushima [1] [2] 80 LeCun (back propagation) LeNet [3, 4] LeNet 2. 2.1 980 8579 6 6 01 okatani@vision.is.tohoku.ac.jp (simple cell) (complex cell)