IPSJ SIG Technical Report Vol.2016-GI-35 No /3/9 StarCraft AI Deep Q-Network StarCraft: BroodWar Blizzard Entertainment AI Competition AI Convo

Size: px

Start display at page:

Download "IPSJ SIG Technical Report Vol.2016-GI-35 No /3/9 StarCraft AI Deep Q-Network StarCraft: BroodWar Blizzard Entertainment AI Competition AI Convo"

ともなりさどひら
4 years ago
Views:

1 StarCraft AI Deep Q-Network StarCraft: BroodWar Blizzard Entertainment AI Competition AI Convolutional Neural Network(CNN) Q Deep Q-Network(DQN) CNN DQN,,, 1. StarCraft: Brood War *1 Blizzard Entertainment ( RTS ) RTS 2010 StarCraft AI Competition * AI BWAPI *3 API StarCraft () () TerranZargProtoss 3 1 HP HP 0 HP 0 StarCraft *1 *2 *3 AI AI Convolutional Neural Network(CNN) Q Deep Q-Network(DQN) AI StarCraft DQN Starcraft 3 Starcraft

2 2. StarCraft StarCraft () () TerranZargProtoss 3 1 Terran SCV Zarg Drone Zarg Terran Creep Creep Protoss Zarg HP HP Pylon HP HP 0 HP Terran Medic Heal Terran Vessel Defensive Matrix 1 HP250 1 StarCraft StarCraft AI StarCraft [1] StarCraft AI 1 AI AI 2

3 α w(z) = Ue iα z w(z) = Q log z(q > 0) w(z) = iγ log z AI [2] 3 AI HP 2 HP r m r t+1 = enemy health it enemy health it+1 i=1 (agent health t agent healt t+1 ) (1) Terran Vulture Marine 6 Vulture 1000 AI 100% RTS AI Kiting 3.4 UCB UCT(UCB applied to Tree) [4] UCT UCB(Upper Confidence Bound) i UCB 3.3 Kiting [3] Kiting one-step Q-learning Watkins s Q(λ)one-step Sarsa Sarsa(λ) UCB(i) = Q i + C ln N N i (2) Q i i C N i N i i C UCB 3

4 i j q (j) i q (j) i = ω 1 HP + ω 2 DM + ω 3 CP + ω 4 EG (3) HP DMCP EG ω n Q i q i 3.5 Deep Q-Network Deep Q-Network Atari 2600 [5] Deep Q-Network Q Q(s, a) CNN Q Experience Replay Replay Memory CNN 2 2 Atari x84 epsilon-greedy Deep Q-Network ( 1 ) Replay-Memory D N ( 2 ) ( 3 ) ( a ) s 1 = {x 1 } ϕ 1 = ϕ(s 1 ) ( b ) t = 1 T ( s T ) ( i ) ϵ a a t ( ii ) Q (ϕ(s t ), a; θ) a t ( iii ) a t r t x t+1 ( iv )s t, a t, x t+1 ϕ(t + 1) ( v ) (ϕ t, a t, r t, ϕ t+1 ) D ( vi )D 1 minibatch 4 (ϕ j, a j, r j, ϕ j+1 ) ( vii )minibatch y j Q r j () y j = b() Deep Q-Network 1000 BreakoutPongEnduro Space Invaders Deep Q-Network *4 StarCraft RTS Deep Q-Network HP DQN CNN CNN *4 distributed-deep-reinforcement-learning/ 4

5 地形情報 CNN ユニット情報 3 DQN Q 学習 DQN 行動 ( 1 ) 32x32 1 8x8 () ( 2 ) CNN ( 3 ) HP Q 9 1 DQN 8 t i reward(i, t) cause damage(i, t) i t unit health(i, t) i t HP unit reward(i, t) =cause damage(i, t) {unit health(i, t) unit health(i, t + 1)} (4) reward(i, t) = 2 unit reward(i, t) unit reward(j, t) (5) 3 j i epsilon-greedy AI 1 DQN 2 1 DQN 4 D enemy (x, y) D enemy (x, y) D enemy (x, y) (D enemy (x, y) ) A* (D enemy (x, y) ) Marine 8 Marine Intel Corei7 6700KPalit NE5XTIX015KB- PG600F (GTX TITAN X 12GB) Windows 10 Starcraft C++ BWAPI 5

6 Python Chainer * 5 MessagePack-RPC * *5 * HP HP AI 10 6

10 1 6. StarCraft DQN DQN AI 2 [1] StarCraft AI 10 (2015). [2] Tung, N., Kien, N. and Ruck, T.

7 StarCraft DQN DQN AI 2 [1] StarCraft AI 10 (2015). [2] Tung, N., Kien, N. and Ruck, T.: Potential flow for unit positioning during combat in StarCraft, IEEE 2nd Global Conference on Consumer Electronics (GCCE 2013), IEEE, pp (2013). [3] Wender, S. and Watson, I.: Applying reinforcement learning to small scale combat in the real-time strategy game starcraft: broodwar, IEEE Conference on Computational Inteligence and Games (CIG 2012),, IEEE, pp (2012). [4] Zhe W., Kien Quang N., Ruck T., Frank R.: MONTE- CARLO PLANNING FOR UNIT CONTROL IN STAR- CRAFT, The 1st IEEE Global Conference on Consumer Electronics 2012, pp (2012). [5] Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D. and Riedmiller, M.: Playing Atari With Deep Reinforcement Learning, NIPS Deep Learning Workshop (2013). 7

2015 3

2015 3 JAIST Reposi https://dspace.j Title ターン制ストラテジーゲームにおける候補手の抽象化によるゲーム木探索の効率化 Author(s) 村山, 公志朗 Citation Issue Date 2015-03 Type Thesis or Dissertation Text version author URL http://hdl.handle.net/10119/12652