情報処理学会研究報告 IPSJ SIG Technical Report Vol.2016-HPC-155 No /8/8 1,a) Convolutional Neural Network (CNN) CNN Stochastic Gradient Descent

Size: px

Start display at page:

Download "情報処理学会研究報告 IPSJ SIG Technical Report Vol.2016-HPC-155 No /8/8 1,a) Convolutional Neural Network (CNN) CNN Stochastic Gradient Descent"

しなつすずがみね
5 years ago
Views:

1 1,a) Convolutional Neural Network (CNN) CNN Stochastic Gradient Descent (SGD) SGD GPU CNN SGD SGD CNN SPRINT CNN TSUBAME-KFC/DL 116 CNN 8% 1. Deep Learning (DL) Deep Neural Network (DNN) [1] [] DNN Stochastic Gradient Descent (SGD, ) DL ( 1) N Minibatch W (t+1) = W (t) η (x ri 0 ; W (t) ) (1) i=1 W (t) t DNN η (x ri 0 ; W ) r i W N Minibatch 1 3 a) oyama.y.aa@m.titech.ac.jp [], [3], [4], [5]. Asynchronous SGD(ASGD)[3], [6] SGD[1], [], [7] ( 1). 1 (t+1) W (t+1) =W (t) ησ i (t) W (t+1) (t+) W (t+) =W (t+1) ησ i (t+1) W (t+) (t+3) W (t+3) =W (t+) ησ i (t+) ASGD GPU E (t) i (t ) GPU ASGD [6]ASGD DNN c 016 Information Processing Society of Japan 1

2 DNN ASGD ASGD DL SPRINT SPRINT GPU Convolutional Neural Network (CNN). DL SPRINT DL SPRINT ILSVRC[8] % top-5 validation error TSUBAME.5 TSUBAME-KFC/DL( 6) 96 GPU.1 SPRINT GPU GPU GPU () All-reduce ( ) SPRINT SPRINT SPRINT : () GPU ( ) mutex Algorithm GPU SSD GPU (1) SSD (N Subbatch ) () η (3) GPU.1. (a) GPU ( ) GPU (b) momentum (c) MPI All-reduce GPU 3. 1 Empirical CNN GPU 1 (Epoch ) GPU CNN c c(c =3) p l = l p l p l = Pooling p l =1 ILSVRC 01, m 0 = 3 (RGB), m L = 1000 (), N File =1, 81, 167 (). c 016 Information Processing Society of Japan

3 1 L L c L x l l (l =0, 1,,L) m 0 m l l (l =1,,,L 1) m L c 3 p l Pooling 1 (l =1,,,L c) N Subbatch 1 N Node N GP U GPU N F ile 3. CNN CNN Max-poolingSoftmax ( 3) [9] inception module[10] CNN. CNN VGG[11] x 0 m 0 0 x 0 3 l 1 x c l 1 m l 1 c x l 1 x Lc m Lc m l p l x Lc x l 1 c +1 p l L c l x l m l x Lc m Lc x l L m L CNN x l (l =1,,,L) x l 1 c+1 p x l = l (l =1,,,L c ) 1 (l = L c +1,L c +,,L) () (Pooling ) x l = x l 1 c +1(l =1,,,L c ), 1(l = L c +1,L c +,,L) CNN () N Param L c L N Param = m l (c m l 1 +1)+ m l (x l 1 m l 1 +1) l=1 l=l c+1 (3) 3.3 SPRINT CUDA cublas cublassgemm imcol convolution CUDA SGEMM { L T ComputeGradient = T cuda (l) l=1 cuda {imcol, } + sgemm {convolution, } } T sgemm (l) (4) CUDA CUDA l imcol l +1( L) 0 αx l c m l 1 N Subbatch + β (l L c ) T imcol (l) = 0 (otherwise) (5) α, β 3.3. SGEMM SGEMM m n k m k k n cublassgemm m, n, k ( ) 4 (a x,a y,a z )(a c 016 Information Processing Society of Japan 3

4 4 cublassgemm () (): m kk n C = AB A B 15 NVIDIA Tesla K80 5 x, y, z ) cublassgemm 8 {mnk,mn,mk,nk,m,n,k} ( 6) T convolution (l) =α mnk c m l 1 m l x l NSubbatch + α mn m l x l NSubbatch + α mk c m l 1 x l NSubbatch + α nk c m l 1 m l + α m x l NSubbatch + α n m l + α k c m l 1 + β (6) SGEMM l convolution 1,,,L c x l N Subbatch m l c m l 1 fc L c +1,L c +,,LN Subbatch m l x l 1 m l 1 dedw L, L 1,, 1 c l 1 m l 1 m l x l N Subbatch dedb L 1,L,, 1 1 m l x l N Subbatch dedx fc L 1,L,,L c N Subbatch x l m l m l+1 dedx conv L c 1,L c,, 1 x l+1 N Subbatch c m l m l+1 Algorithm 1 GPU t 1: repeat : LatestWeights 3: if LatestWeights then 4: Weights LatestWeights 5: end if 6: LatestWeights 7: SSD 8: GPU 9: Grad (Weights ) 10: Grad ηgrad 11: GradBuf t 1: if GradBuf t then 13: GradBuf t Grad 14: else 15: GradBuf t GradBuf t + Grad 16: end if 17: GradBuf t 18: until 3 GPU LockWeights G TUpdateW eights TUpdate (N GP U 1) FetchWeights 3-5 αn Param + TF etchw eights TGP U min(t GP U /T Update, 1) LoadImage 7 αn Subbatch + β DeformImage 8 αn Subbatch + β ComputeGradient 9 Equation 4 ComputeUpdateVal 10 αn Param LockGradient G 11 UpdateGradient 1-16 αn Param T waiting = t T E(T waiting for t ) (TSumGradient/NGP U ) TUpdate = t T E(T waiting for t t CS ) = t T = t T P (t CS ) T t critical section T t critical section T t iteration T t critical section T t iteration (7) T T t critical section 3.4 GPU GPU t (t =1,,,N GP U ) Algorithm 1 8 ( 3) α, β (LockWeights GLockGradient G) (CS) FCFS t T t iteration t. FetchWeights LatestWeights GPU T GP U /T Update (T GP U <T Update ) 1() 3.5 Algorithm (x) n x n (n =1,,,N Node ) c 016 Information Processing Society of Japan 4

5 1/N Node GPU 8 ( 4) Algorithm 1: repeat : for t =1toN GP U do 3: GradBuf t 4: if GradBuf t then 5: if t =1then 6: SendBuf GradBuf t 7: else 8: SendBuf SendBuf + GradBuf t 9: end if 10: else if t =1then 11: SendBuf 0 1: end if 13: GradBuf t 14: end for 15: OldWeights (LatestWeights) n 16: SendBuf SendBuf+(OldWeights) n +ν(deltaweights) n 17: RecvBuf MPI Allreduce(SendBuf) 18: DeltaWeights (RecvBuf) n OldWeights 19: LatestWeights 0: LatestWeights RecvBuf 1: LatestWeights : until 4 LockGradient U 3 N GP U TUpdateGradient TGP U SumGradient 4-1 αn GP U N Param UpdateOldWeights 15 min(t Update /T GP U, 1) αn Param /N Node AddMomentum 16 αn Param /N Node Allreduce 17 T Barrier + UpdateMomentum 18 (α log (N Node )+β) N Param αn Param /N Node LockWeights U 19 N GP U UpdateWeights 0 αn Param TF etchw eights TGP U SumGradient GPU UpdateWeights Allreduce Allreduce 1 GradBuf t SendBuf X i B(N GP U,p)(p =min(t Update /T GP U, 1)) X M = max(x 1,X,,X NNode ) ( 8) T Barrier = α E(X M X 1 ) N GP U 1 } = α {N GP U (1 p) F (i) NNode i=0 (8) F (i) B(N GP U,p) p =1 N Node =1 T Barrier =0 3.6 GPU (N Node N GP U N Subbatch )/T GP U N Minibatch avg = N Node N GP U N Subbatch T Update T GP U (9) Epoch T Epoch = N File T Update 4. = N Minibatch avg N File T GP U N Node N GP U N Subbatch (10) SPRINT TSUB- AME.5TSUBAME-KFC/DL ILSVRC 01 CNN CNN-A, CNN-B, CNN-C ( 5) 5 CNN x 0 L c L #{l p l > 1} N Param (10 6 ) CNN-A CNN-B CNN-C FLOP/s Tesla K80 GK10 GPU TSUBAME-KFC/DL GPU N GP U =8 4.1 CUDA CNN-A( ) TSUBAME.5 TSUBAME-KFC/DL 5 N Subbatch TSUBAME.5 N Subbatch = 1,,, 5TSUBAME-KFC/DL c 016 Information Processing Society of Japan 5

6 6 TSUBAME.5 TSUBAME-KFC/DL CPU Intel Xeon CPU X5670 IntelXeonE5-60v.93 GHz, 6.1 GHz, 6 54 GB DDR3 64GB DDR3 GPU NVIDIA Tesla K0X 3 NVIDIA Tesla K TFLOP/s 8.74 TFLOP/s 6GBGDDR5 4 GB GDDR5 ECC ECC Auto boost SSD HP B1 Intel SSDSCBB480G4 60GB SATA3 10GB SATA3 4X QDR InfiniBand 4X FDR InfiniBand 4GB/s 7GB/s OS SUSE Linux Enterprise CentOS 6.4 Server 11 SP3 icpc 14.0 icpc CUDA CUDA 7.0 CUDA 7.0 MPI MVAPICH.0rc1 MVAPICH.0rc1 pthread NPTL.11.3 NPTL.1 N Subbatch =1,,, 6 5 CUDA 5 TSUBAME-KFC/DL CUDA () (): 5 5%x CUDA (imcol 1013 MBsoftmax 3.4 KB ) SGEMM (m, n, k) =( x, y, z )(x, y, z ) (5 ) (m, n, k) =( x+1 4, y+1 4, z+1 4 ) % 4 CNN-A, CNN-B, CNN-C % 10 % 0 % 30 % 40 % 50 % 60 % 70 % 80 % 90 % 100 % 110 % 6 TSUBAME-KFC/DL cublassgemm 7 5 SPRINT N Subbatch CNN N Subbatch = 6 7 CNN-A dedw() N Subbatch =6 scal kernel sgemm largek lds64 (cublassgemm ) N Subbatch = 7 sgemm sm35 ldg tn 3x16x64x8x N Subbatch SGEMM 7 TSUBAME-KFC/DL() TSUBAME.5() () () CNN-C N Subbatch =4 8 SGEMM CUDA cublassgemm 7 1% 7 TSUBAME.5 TSUBAME-KFC/DL [%] [%] [%] [%] CNN-A CNN-B CNN-C c 016 Information Processing Society of Japan IPSJ SIG Technical Report 6

7 8 TSUBAME-KFC/DL CNN-C N Subbatch =4 10 TSUBAME-KFC/DL 4. CNN-A TSUBAME.5: N GP U =3,N Node =, 4, 8,, 3, N Subbatch =1,,, 5, 10 5 TSUBAME-KFC/DL: N GP U =8,N Node =, 4, 8, N Subbatch =1, 4, 8, 11, TSUBAME.5 T Update (N Node,N Subbatch ) N GP U = 8, N Node = 1,, 4,, 16, N Subbatch = 1, 4, 8, 11 CNN-A GPU T GP U N Subbatch 10 T Update N Node N Subbatch N Node =8,N Subbatch =4 11 MPI T Allreduce T SumGradient SPRINT 1CPU 9 TSUBAME-KFC/DL GPU : (N Node,N Subbatch ) TSUBAME.5 N GP U = 3, N Node = 1,, 4,, 64, N Subbatch = 1,,, 5 11 TSUBAME-KFC/DL N Node =8,N Subbatch =4 T GP U, T Update, T Epoch, N Minibatch avg 8 T Epoch T GP U T GP U 8 CNN-A TSUBAME.5 TSUBAME-KFC/DL [%] [%] [%] [%] T GP U T Update T Epoch N Minibatch avg CNN- B(N Node =1,, 4, 8, 16, N Subbatch =1, 3, 5) CNN-C (N Node =1,, 4, 8, 16,N Subbatch =1, 4) ( 9) 9 TSUBAME-KFC/DL CNN-BCNN-C CNN-B CNN-C [%] [%] [%] [%] T GP U T Update T Epoch N Minibatch avg TSUBAME-KFC/DL CNN-A, CNN-B, CNN-C T Epoch,N Minibatch avg 6%8% c 016 Information Processing Society of Japan 7

8 4.3 ASGD DL N Node,N Subbatch TSUBAME.5 TSUBAME-KFC/DL CNN-A T Epoch N Minibatch avg ( 1 13) 138 ± 5% N Minibatch avg = 138 TSUBAME.5 N Node = 16, N Subbatch =5 N_Subbatch N_Subbatch e+0 sec 1e+03 sec e+03 sec 5e+03 sec 1e+04 sec e+04 sec 5e+04 sec 1e+05 sec e+00 5e+00 1e+01 e+01 5e+01 1e+0 e+0 5e+0 1e N_Node 1 TSUBAME.5 T Epoch () N Minibatch avg () : N Minibatch avg 138 ± 5% T Epoch N_Subbatch e+0 sec 1e+03 sec e+03 sec 5e+03 sec 1e+04 sec e+04 sec 5e+04 sec 1e+05 sec T Epoch,N Minibatch avg : T TSUBAME.5 KFC TSUBAME- KFC/DL T Epoch T Epoch [sec] N Minibatch avg N Node N Subbatch [%] [%] KFC KFC T T T T Epoch SPRINT [5] DL Rudra Rudra learner () staleness staleness 6. N_Subbatch e+00 5e+00 1e+01 e+01 5e+01 1e+0 e+0 5e+0 1e N_Node 13 TSUBAME-KFC/DL T Epoch () N Minibatch avg () T Epoch N Minibatch avg [1] CPU DL SGD CNN SPRINT TSUBAME-KFC/DL 116 CNN 8% EBD c 016 Information Processing Society of Japan 8

9 [1] Wu, R., Yan, S., Shan, Y., Dang, Q. and Sun, G.: Deep Image: Scaling up Image Recognition, CoRR, Vol. abs/ (online), available from (015). [] Amodei, D., Anubhai, R., Battenberg, E., Case, C., Casper, J., Catanzaro, B., Chen, J., Chrzanowski, M., Coates, A., Diamos, G., Elsen, E., Engel, J., Fan, L., Fougner, C., Han, T., Hannun, A., Jun, B., LeGresley, P., Lin, L., Narang, S., Ng, A., Ozair, S., Prenger, R., Raiman, J., Satheesh, S., Seetapun, D., Sengupta, S., Wang, Y., Wang, Z., Wang, C., Xiao, B., Yogatama, D., Zhan, J. and Zhu, Z.: Deep Speech : End-to-End Speech Recognition in English and Mandarin, ArXiv e- prints (015). [3] Zhang, S., Zhang, C., You, Z., Zheng, R. and Xu, B.: Asynchronous stochastic gradient descent for DNN training, Acoustics, Speech and Signal Processing (ICASSP), 013 IEEE International Conference on, pp (online), DOI: /ICASSP (013). [4] Krizhevsky, A.: One weird trick for parallelizing convolutional neural networks, CoRR, Vol. abs/ (online), available from (014). [5] Gupta, S., Zhang, W. and Milthorpe, J.: Model Accuracy and Runtime Tradeoff in Distributed Deep Learning, ArXiv e-prints (015). [6] Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P., Yang, K., Le, Q. V. and Ng, A. Y.: Large Scale Distributed Deep Networks, Advances in Neural Information Processing Systems 5 (Bartlett, P., Pereira, F., Burges, C., Bottou, L. and Weinberger, K., eds.), pp (online), available from pdf (01). [7] Iandola, F. N., Ashraf, K., Moskewicz, M. W. and Keutzer, K.: FireCaffe: near-linear acceleration of deep neural network training on compute clusters, CoRR, Vol. abs/ (online), available from (015). [8] Stanford Vision Lab, Stanford University, P. U.: ImageNet. available from [9] Srivastava, R. K., Greff, K. and Schmidhuber, J.: Highway Networks, CoRR, Vol. abs/ (online), available from (015). [10] Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V. and Rabinovich, A.: Going Deeper With Convolutions, The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (015). [11] Simonyan, K. and Zisserman, A.: Very Deep Convolutional Networks for Large-Scale Image Recognition, CoRR, Vol. abs/ (online), available from (014). [1] Yan, F., Ruwase, O., He, Y. and Chilimbi, T.: Performance Modeling and Scalability Optimization of Distributed Deep Learning Systems, Proceedings of the 1th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 15, New York, NY, USA, ACM, pp (online), DOI: / (015). c 016 Information Processing Society of Japan 9

untitled

untitled c ILSVRC LeNet 1. 1 convolutional neural network 1980 Fukushima [1] [2] 80 LeCun (back propagation) LeNet [3, 4] LeNet 2. 2.1 980 8579 6 6 01 okatani@vision.is.tohoku.ac.jp (simple cell) (complex cell)