10E3 Automatic Adjustment of Controller Parameters Using Reinforcement Learning *K. Yamamoto, F. Kawamoto, M. Nishida and Y. Wakasa Yamaguchi Universi

Size: px

Start display at page:

Download "10E3 Automatic Adjustment of Controller Parameters Using Reinforcement Learning *K. Yamamoto, F. Kawamoto, M. Nishida and Y. Wakasa Yamaguchi Universi"

つかさほがり
1 years ago
Views:

1 10E3 Automatic Adjustment of Controller Parameters Using Reinforcement Learning *K. Yamamoto, F. Kawamoto, M. Nishida and Y. Wakasa Yamaguchi University Abstract Fictitious Reference Iterative Tuning (FRIT) is a practical controller tuning method that does not require iterative experiments. However, in FRIT, it is necessary for a control system designer to set a reference model. Therefore, in this paper, we propose a method for determining a reference model using a neural network learned by reinforcement learning. Reinforcement learning is a kind of machine learning method that learns the optimal answer by trial and error. By enabling the neural network to determine the reference model, the control system designer does not need to set the reference model. Also, a response prediction method is used in order to take advantage of the FRIT s superiority that the repetitive experiment is unnecessary. Key Words: Controller tuning, Machine learning, Reinforcement learning 1 Fictitious Reference Iterative Tuning (FRIT) [1] [2] Neural Network (NN) NN [3] NN NN (deep learning) NN NN Recurrent Neural Network (RNN) NN RNN Long Short Term Memory (LSTM) [3] FRIT [4] FRIT FRIT [5] FRIT 2 FRIT FRIT [1, 6]PID Fig. 1 Fig. 1 G(z) 1 1 u(k) r (k) + e (k) u(k) y(k) C( z, θ) G(z) Fig. 1: Closed-loop system 第 61 回自動制御連合講演会 (2018 年 11 月 17 日 ~ 18 日, 名古屋 ) 1538

2 y(k) PID C(z, θ) r(k) M(z) PID C(z, θ) T(z, θ) = G(z)C(z, θ) 1 + G(z)C(z, θ) (1) r + Environment Control system Controller Plant Controller tuning by FRIT Reference model M Action a t y PID θ 0 θ 0 PID r(k) u(θ 0, k) y(θ 0, k) Fig. 1 u 0 (k) y 0 (k) r(θ, k) r(θ, k) = C(z, θ) 1 u 0 (k) + y 0 (k) (2) J(θ) = y 0 (k) M(z) r(θ, k) 2 (3) J(θ) (y 0 (k), M(z), r(θ, k)) (3) J(θ) PID 1 FRIT 3 2 FRIT M(z) Asynchronous Advantage Actor-Critic (A3C) [7] NN 2 NN FRIT NN Agent (NN) Response of control system y Fig. 2: Framework of the proposed controller adjustment 3.1 FRIT reinforcement learning [8] R FRIT Fig. 2 y(k) M(z)r(k) 2 R e = N k=1 (M(z)r(k) y(k)2 ) y(k) R r R o s t R = W e (1 R e R e + 20 )+W r ( 1 R r R r + 5 ( )+W o 1 R ) o R o + 2 Fig. 2 NN NN M(z, p t ) p t NN t Fig. 2 y s t a t a t p t+1 s t R e R r R o p t s t Q-learning Actor-Critic [9] A3C [7] A3C 1539

3 Thread 1 Thread 2 Thread n Environment Control system FRIT Reference model M action Response Agent (NN) w w w w Global Neural Network... w w START 初期設定 ( 学習全体 ) パラメータ p の設定 ( ランダム ) FRIT による制御器調整状態 s t の取得ローカル NN パラメータ初期化 t start = t Block1 NNにより行動選択パラメータpの設定 t t + 1 Block2 A 勾配の蓄積 i = t 1,, t start グローバルNNパラメータの更新 s t = s end t = 1, T = T + 1 Fig. 3: configuration of the A3C NN Asynchronous Fig. 3 CPU NN NN w NN w NN V Temporal Difference (TD) A3C Advantage Advantage 1 k Actor-Critic Actor Critic Actor Critic Actor Fig. 4 T max Advantage t max 1 s end T = 0, t = 1 p 1 G 1 2 FRIT M(z, p t ) s t a t (p t+1 p t + p t ) p t+1 p t NN M(z, p t+1 ) FRIT による制御器調整状態 s t の取得 t t start = t max A T = T max END Fig. 4: Flowchart of the lerning process FRIT s t+1 FRIT Fig. 4 Block1 Block2 FRIT t max NN 1 A3C Advantage Fig. 3 NN NN NN NN G 1 G n 1 n NN NN FRIT NN 1540

4 e(k) unit u(k) Forget gate Input gate z -1 Output gate Memory cell tanh σ σ tanh σ xk z -1 v k Fig. 5: Neural network Fig. 6: LSTM block Fig. 2 NN FRIT FRIT NN NN NN NN Fig. 2 G 1 G n FRIT FRIT FRIT FRIT s t FRIT Neural Network Fig. 5 NN Fig. 5 RNN 10 LSTM [3] LSTM Fig. 6 LSTM LSTM 4 [0, 1] Fig. 6 tanh, σ x k v k 3.2 [5] C ϕ T(ϕ) T(ϕ) T(ϕ) = ϕ (m+n+1)s m + + ϕ n+2 s + ϕ n+1 ϕ n s n + + ϕ 1 s + 1 (4) ϕ := [ϕ 1 ϕ 2 ϕ m+n+1 ] u 0 y 0 C ϕ J(ϕ) = y 0 (k) T(ϕ) r 2 (5) r (2) (5) J(ϕ) ϕ C C T(ϕ ) T(ϕ ) r C y p (k) 1541

5 4 1 (p t s+1) 2 M(z, p t ) T max = 1200, t max = 3, L u = 2.5, L l = 0.1 s end t = 10 p t p t = L u p t = L l W e = 0.6, W r = 0.35, W o = 0.05 NN Actor Critic 4 90% 1 10 LSTM Actor 30 Critic 1 CPU 1 5 G 1 G 5 FRIT PID θ 0 = [K P, K I, K D ] T = [0.5, 0.5, 0] T 0.01 s r(k) Python Chainer G(s) = (12s + 8)/(20s s s s + 1) [2] M(z, p t ) FRIT y p (k) M(z, p t )r(k) T(ϕ) = ϕ 5 ϕ 4 s 4 + ϕ 3 s 3 + ϕ 2 s 2 + ϕ 1 s + 1 (6) NN s t p t FRIT PID θ 0 = [K P, K I, K D ] T = [0.5, 0.5, 0] T 1 FRIT p 1 = 1 FRIT p Fig. 7 Fig. 8 FRIT 1 Fig. 9 Fig. 11 Fig p t 0.66 Fig Parameter p Reward Fig. 7: Reference model parameter Fig. 8: Reward Fig. 9 Fig FRIT Fig FRIT 100 FRIT NN FRIT 5 FRIT [1],, : ; 1542

6 Error 0.15 Overshoot Fig. 9: Root mean square error Fig. 11: Overshoot Rise time Output Output of the proposed method Output after the first FRIT Reference signal Fig. 10: Rise time, Vol. 17,. 12, pp (2004) [2] : ;, Vol. 47,. 11, pp (2008) [3] :, (2015) [4],, : ; 35, pp. 165 (2018) [5], : 2 -ERIT -; 5, Fr72-3 (2018) Time [s] Fig. 12: Control output [9] V. Mnih, K. Kavukcuoglu, D. Silver, A. Rusu, J. Veness, M. Bellemare, A. Graves, M. Riedmiller, A. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg and D. Hassabis: Human-level control through deep reinforcement learning, Nature, Vol. 518, pp (2015) [6],,,, : PID ;, Vol. 22,. 4, pp (2009) [7] V. Mnih, A. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver and K. Kavukcuoglu: Asynchronous methods for deep reinforcement learning, Proceedings of the 33rd International Conference on Machine Learning, Vol. 48, pp (2016) [8] :,, Vol. 52,. 1, pp (2013) 1543

DQN Pathak Intrinsic Curiosity Module (ICM) () [2] Pathak VizDoom Super Mario Bros Mnih A3C [3] ICM Burda ICM Atari 2600 [4] Seijen Hybrid Reward Arch

DQN Pathak Intrinsic Curiosity Module (ICM) () [2] Pathak VizDoom Super Mario Bros Mnih A3C [3] ICM Burda ICM Atari 2600 [4] Seijen Hybrid Reward Arch Hybrid Reward Architecture 1,a) 1 AI RPG (Rogue-like games) AI AI A3C ICM ICM Deep Reinforcement Learning of Roguelike Games Using Internal Rewards and Hybrid Reward Architecture Yukio Kano 1,a) Yoshimasa