DEIM Forum 2018 C ARIMA Long Short-Term Memory LSTM

Size: px

Start display at page:

Download "DEIM Forum 2018 C ARIMA Long Short-Term Memory LSTM"

ひろとしもとり
5 years ago
Views:

1 DEIM Forum 2018 C ARIMA Long Short-Term Memory LSTM Bayesian Optimization [1] [2] Multi-Task Bayesian Optimization 1 LSTM [3] [1] Branin-Hoo [4] MNIST CIFAR-10 [5] Convolutional neural networks: CNN [2] Support Vector Machine SVM LSTM Multi-Task Gaussian Process [1] Meta feature [2] 2. ARIMA Deep Belief Network(DBN) LSTM 2. 1 ARIMA [6] y t y t = y t y t 1 I(1)

2 ARMA MA AR MA AR ARMA MA MA(q) y t = µ + ε t + θ 1ε t 1 + θ 2ε t θ qε t q, (1) ε W.N.(σ 2 ) µ θ W.N. ε σ 2 MA(q) q MA(q) q AR AR p AR AR(p) y t = c + ϕ 1y t 1+ϕ 2y t ϕ py t p + ε t, (2) ε W.N.(σ 2 ) c ϕ AR 2 ARMA y t = c + ϕ 1y t 1+ϕ 2y t ϕ py t p + ε t + θ 1ε t 1 + θ 2ε t θ qε t q, (3) ε W.N.(σ 2 ) AR MA ARMA 2. 2 DBN(Deep Belief Network) DBN Restricted Boltzmann Machine(RBM) DBN RBM(Restricted Boltzmann Machine) RBM(Restricted Boltzmann Machine) 1 1 Deep Belief Network(DBN). 2 2 n v 2 v n h 2 h RBM E(v, h) P (v, h) = 1 exp{ E(v, h)} (4) Z E(v, h) = b T v c T h v T W h (5) Z = exp{e(v, h)} v (6) h Z v, h P (v) RBM P (h v) P (v h) Contrastive Divergence CD P (h v) P (v h) [7] Contrastive Divergence [8] DBN(Deep Belief Network) DBN Deep Belief Network [7] Hinton DBN RBM DBN 2 DBN 2 1 DBN

3 P (h (l), h (l 1) ) (7) exp (b (l)t v (l) + b (l 1)T h (l 1) + v (l 1)T W (l) h (l)) ( P (h (k) i = 1 h (k+1) ) = σ b (k) i + W (k+1)t :,i h (k+1)) (8) i, k 1,..., l 2 ( P (v i = 1 h (1) ) = σ b (0) i + W (1)T :,i h (1)) (9) 3 β ( v N v; b (0) + W (1)T h (1), β 1) (10) DBN 2 DBN CD 1 RBM E v pdata log p(v) 2 RBM W in W out W LSTM Long Short-Term Memory 10 LSTM [3] 3 E v pdata E h (1) p (1) (h (1) v) log p(2) (h (1) ) (11) p (1), p (2) 1 2 RBM 2 RBM 1 DBN DBN [7] MLP DBN Recurrent Neural Network RNN [9] 2 2 Recurrent Neural Network(RNN). 3 LSTM memory units. Input Gate Input MeMory Cell Output Gate Memory Cell Output f 1 n D n = {x n, y n} N n=1 x n = {x 1,..., x m} m y n f loss accuracyvalidation error n

4 n + 1. acquisition function α(x; D n) n + 1 x n+1 x n+1 = argmax α(x; D n) (12) x [10] x n+1 f y n+1 D n D n y n+1 [11] p(d n (x n+1, y n+1)) p((x n+1, y n+1) D n+1) n + 1 D n+1 p((x n+1, y n+1) D n+1) = p(xn+1, yn+1)p(dn (xn+1, yn+1)) p(d n) (13) x n+1 y n+1 D n+1 M 1 Algorithm 1 Bayesian Optimization for n = 1, 2,..., do select new x n+1 by maximize acquisition function α x n+1 = argmax x α(x; D n) obtain y n+1 add data D n+1 by use Bayes theorem p((x n+1, y n+1 ) D n+1 ) = p(x n+1,y n+1 )p(d n (x n+1,y n+1 )) p(d n) update probabilistic model M end for D n D n x Multi-task Bayesian Optimization, [2] Meta Feature [1] [2] [2] D N+1 D N+1 D i(i = 1,..., N) D N+1 x D N+1 [2] 2 1 L p L p d(d i, D j) = m i m j p (14) p = 1 p = 2 D i(i = 1,..., N) m i = (m i 1,..., m i n) [2] D i, D j n D i, D j F D i = [f D i (x 1),..., f D i (x n)] F D j = [f D j (x 1),..., f D j (x n)] d(d i, D j) = 1 Corr(F D i, F D j ) Corr(F D i, F D j ) = 6 n k=1 d2 k n 3 n (15) d k F D i k f D i (x k ) F D j k f D j (x k ) D n+1 f D N+1 (x 1),..., f D N+1 (x n) D n+1 m i, m i d(d i, D j) R d(d N+1, D j) R(m N+1, m j ) (16) x n(n = 1,..., N) D n(n = 1,..., N)

5 d D N+1 D i(i = 1,..., N) ϕ(i) = d(d N+1, D i) ϕ(i) t x ϕ(1),..., x ϕ(t) t = 1 2 Algorithm 2 Initialization sort dataset by increasing distance to D N+1 i.e.: (ϕ(i) < = ϕ(j)) (d(d N+1, D i ) < = d(d N+1, D j )) for i = 1,..., t do x i = x ϕ(i) end for x = BayesianOptimization( x 1,..., x t ) return x 3. 3 LSTM [2] (1). LSTM 1 ϵ ρ 2 LSTM RMSprop RMSprop h t = αh t 1 + (1 ρ) E(w t ) 2 η0 η t = ht + ϵ w t+1 =w t η t E(w t ) (17) w η RMSprop [12] LSTM LSTM LSTM LSTM LSTM Deep-LSTM Dropout LSTM 4. (2). (3). 2. (4). 4 accuracy relational task 3 objective task LSTM WIND database S&P 500 United States 1923 Nikkei 225 Japan 1950 Hang Seng Hong-Kong 1969 CSI 300 China 2005 Nifty 50 India GPS

4 The outline of Multi-Task Bayesian Optimization for LSTM. 1 1 Description of the input variables. MACD CCI ATR BOLL EMA20 MA5/MA10 MTM6/MTM12 SMI ROC WVAD Closed Price 1 4.

6 4 The outline of Multi-Task Bayesian Optimization for LSTM. 1 1 Description of the input variables. MACD CCI ATR BOLL EMA20 MA5/MA10 MTM6/MTM12 SMI ROC WVAD Closed Price [13] Diagram of LSTM for financial time series [13]. LSTM LSTM LSTM LSTM LSTM DBN LSTM MAPE 100 n n fi yi y i (18) k=1 LSTM MAPE MAPE

7 3 3 MAPE 2 2 Optimization results for all indexes in validation data. Multi-task BO Single-task BO Random Search S&P Nikkei Hang seng CSI Nifty Nikkei 225 2% 3 3 Parameter settings. S&P 500 Nikkei 225 Hang seng CSI 300 Nifty 50 U La Le B E ρ ϵ 1.90E E E E E-8 D MAPE A BO MTBO BO MTBO BO U La Le B E D A Nikkei 225 CSI 300 Hang seng Nifty Hang Seng ρ 1 S&P 500 Nikkei 225 A Hang Seng CSI 300 B A B A B S&P 500 MAPE 3 MAPE 4 4 Optimization results for all indexes in test data. MTBO BO RS avg. med. s.d avg. med. s.d avg. med. s.d NK HS CS NF NK Nikkei 225 HS Hang Seng CS CSI 300 NF Nifty 50 CSI 300 CSI 300

8 4 s.d MAPE Nifty Prediction result of all optimization ways in Nifty Nifty 50 1 Hang Seng CSI 300 Nikkei 225 S&P CSI 300 S&P 500 Hang Seng Nikkei CSI 300 Nikkei 225 Hang Seng S&P 500 MAPE 5 5 Multi-task Bayesian Optimization results for random order of indexes. pattern 1st 2nd 3rd 1 HS: CS CS CS: SP NK NK: HS HS SP: NK SP NF: NF NF SP S&P 500 CSI 300 S&P 500 Nikkei LSTM [1] LSTM. (B) (15H02703) [1] Kevin Swersky, Jasper Snoek, Ryan P. Adams Multi-Task Bayesian Optimization Advances in Neural Information Processing Systems 26 (2013) [2] Matthias Feurer, Jost Tobias Springenberg, Frank Hutter Initializing Bayesian Hyperparameter Optimization via Meta-Learning Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence AAAI 15 (2015) [3] Hochreiter, Sepp and Schmidhuber, Jürgen Long Short- Term Memory Neural Comput.(1997) [4] Donald R. Jones A taxonomy of global optimization methods based on response surfaces Journal of Global Optimization(2001) [5] Alex Krizhevsky Learning multiple layers of features from tiny images Technical report, Department of Computer Science, University of Toronto,(2009) [6] (2010) [7] Ian Goodfellow, Yoshua Bengio, Aaron Courville Deep Learning, chapter 20 MIT Press(2016) [8] Hinton, Geoffrey E. Training Products of Experts by Minimizing Contrastive Divergence Neural Comput.(2002) [9] Ian Goodfellow, Yoshua Bengio, Aaron Courville Deep Learning, chapter 10 MIT Press(2016) [10] Jasper Snoek; Hugo Larochelle; Ryan P. Adams Practical Bayesian Optimization of Machine Learning Algorithms Advances in Neural Information Processing Systems 25 (2012) [11] Carl E. Rasmussen and Christopher Williams Gaussian Processes for Machine Learning MIT Press,(2006) [12] T Tieleman, G Hinton Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude COURSERA: Neural networks for machine learning(2012) [13] Bao W, Yue J, Rao Y A deep learning framework for financial time series using stacked autoencoders and long-short term memory. PLoS ONE 12(7): e (2017)

Convolutional Neural Network A Graduation Thesis of College of Engineering, Chubu University Investigation of feature extraction by Convolution

Convolutional Neural Network A Graduation Thesis of College of Engineering, Chubu University Investigation of feature extraction by Convolution Convolutional Neural Network 2014 3 A Graduation Thesis of College of Engineering, Chubu University Investigation of feature extraction by Convolutional Neural Network Fukui Hiroshi 1940 1980 [1] 90 3