2017 (413812)

Size: px

Start display at page:

Download "2017 (413812)"

しまなまつかた
5 years ago
Views:

1 2017 (413812)

2 Deep Learning ( NN) 2012 Google ASIC(Application Specific Integrated Circuit: IC) 10 ASIC Deep Learning TPU(Tensor Processing Unit) NN

3 Abstract Multi-layered neural network(nn) has been announeced in 2012 and it can provide a high recognition performance by using Deep Learning. in recent years, many companies and researchers are actively researching application for image recognition, such as vehicle detection, pedestrian recognition, and speech recognition. Especially, Google has been developed the TPU(Tensor Processing Unit) by using ASIC(Application Specific Integrated Circuit), which is a dedicated processor for Deep Learning. The increase of the performance per power consumption for the TPU is ten times that of the conventional method. However, in order to improve the performance cost ratio of this dedicated hardware, it is essential to reduce the number of required transistors while suppressing the computation accuracy to improve the performance per power consumption. In this study, we proposed a method that switch over the operation precision of the multiplier during learning or switch over the operation precision of the multiplier that occupy the majority of constituent gates for each layer of NN. The evaluation results show that the performance of the operation with 12bit are improved by 20 to 30 than the previous study.

4 Deep Learning Deep Learning Deep Learinig GPU ASIC A 26 i

5 B 28 ii

6 iii

7 5.1 MNIST MNIST ( ) ( ) iv

8 1 1.1 ( NN) Deep Learning NN 1940 NN NN 2012 ILSVRC2012(ImageScaleVisual RecognitionChallenge) FisherVector Deep Learning Google YouTube 1 Deep Learning Deep Learning 1

9 Google Facebook,Deep Learning Google Deep Learning TPU(Tensor Processing Unit) [1] TPU Deep Learning ASIC(Application Specific Integrated Circuit: IC) GPU CPU 10 TPU GPU 1.2 Deep Learning NN GPU 3 Deep Learning 2

10 Deep Learning GPU 8bit Deep Learning GPU FPGA TPU FPGA 16bit [2] 16bit FPGA GPU 2 Deep Learning NN

11 NN NN 2006 NN Deep Learning NN NN NN NN 2.1 NN 2.1 y = f( i w i x i + b) 4

12 x w b f(x) = exp( x) (1) f(x) = tanh(x) (2) f(x) = max(0, x) (3) 2.1: 2.2 (NN) NN 5

13 : 1 6

14 2.3: 2.3 NN(CNN:Convolutional Neural Network) 2.4: CNN 7

15 softmax n x n y n w n w n x = n x n w + 1 n y = n y n w + 1 8

16 2.5: h i = 1 P i Lp h i = ( 1 P i j P i h j (4) h i = max j P i h j (5) h P j ) 1 P (6) j P i k 1 P i k i k 1 P i k i 9

17 Lp 2.6: 2.4 Deep Learning Deep Learning NN Deep Learning ( ) C = i d i log p i (7) d i C 10

18 2.4.1 w ij w ij w ij + w ij = ϵ C w ij (8) C ϵ C C w ij C w ij C w ij = (p i d i )h j (9) 11

19 f(f(f)) 3 l i j i x i = j w ij h j (10) w ij C w ij C = C x i (11) w ij x i w ij 1 δ i C x i 2 x i = j w ij h j x i w ij = h j (12) l x l x l = i w li h i = i w li f(x i ) x i C x l (l = 1 2 ) δ i δ i = C x i = l C x l x l x i (13) 12

20 1 δ l C x l 2 x l = i w li f(x i ) x l x i = f (x i )w li (14) δ i = f (x i ) l δ l w li (15) δ l δ i δ l δ l = p l d l (15) δ i δ i (11) C w ij C w ij = δ i h j (16) δ i δ i 13

21 2.7: 2.8: 14

22 3 Deep Learinig Deep Learning GPU GPU 3.1 GPU GPU NN GPU GPU NN [3] 15

23 3.2 ASIC ASIC Application Specific Integrated Circuit ASIC GPU Google Apple Google AI AlphaGo Apple ios ASIC GPU ASIC ASIC 16

24 4 CNN(Convolutional Neural Network) 16bit 12bit 12bit 4.1 NN 16bit NN 17

25 4.9: epoch epoch 18

26 4.10: 4.3 Python GPU Deep Learning Theano Deep Learning Tutorials GPU 1bit 19

27 4.4 IT IP Deep Learning Deep Learning 12bit 16bit 12bit 16bit 12bit 10 20

28 5 5.1 MNIST MNIST * MNIST CONV POOL FULLY SOFTMAX RELU M i R i C i M o R o C o K r K c K r K c R i C i R o C o 5.1: MNIST MNIST M i, R i, C i K r, K c M o, R o, C o INPUT - - 1,28,28 CONV 1,28,28 5,5 20,24,24 POOL 20,24,24 2,2 20,12,12 CONV 20,12,12 5,5 50,8,8 POOL 50,8,8 2,2 50,4,4 FULLY 800,-, ,-,- SOFTMAX 500,-,- - 10,-,- 21

29 5.2 (8 ) bit bit 16bit 1 [4] bit 10 12bit 16bit GPU 4 16bit 3 12bit 22

30 5.2: MNIST bit CONV RELU SOFTMAX 12 CONV RELU SOFTMAX CONV RELU 16 SOFTMAX 12 CONV RELU 12 SOFTMAX 12 CONV 12 RELU SOFTMAX 12 CONV 16 RELU 16 SOFTMAX : ( ) bit bit : ( ) bit bit

31 6 16bit 12bit 12bit 10 NN CNN NN Deep Learning 24

32 [1], Google TPU GPU 10, jp/atcl/ncd/14/457163/ / (2016/12/10 ) [2] S.Fupta,et al., Deep Learning with Limited Numerical Precision,ICML-15,pp ,2015 [3] NVIDIA, images.nvidia.com/content/apac/events/deep-learning-day jp/nvidia-deeplearning-intro.pdf (2017/1/30 ) [4],,,

33 A ########################### #5 ######################### if this_validation_loss >= best_validation_loss: count += 1 count2 = 0 print "best_error %.3f, this_error %.3f, count %d count12 % if count3 == 1: count16 +=1 if count3 == 0: count12 +=1 else: count = 0 count2 +=1 print "best_error %.3f, this_error %.3f, count %d count12 % if count3 == 1: count16 +=1 26

34 if count3 == 0: count12 +=1 if count == 5: conv_digit = 4 relu_digit = 4 print "digit updated" count3 = 1 ########################### #3 ######################### if count2 == 3: if count3 == 1: if count4 == 0: conv_digit = 3 relu_digit = 3 print "digit rollback" count3 = 0 count4 = 1 27

35 ############################# B MNIST 28

Deep Learning Deep Learning GPU GPU FPGA %

2016 (412825) Deep Learning Deep Learning GPU GPU FPGA 16 1 16 69% Abstract Recognition by DeepLearning attracts attention, because of its high recognition accuracy. Lots of learning is necessary for Deep