,.,., ( ).,., A, B. A, B,.,...,., Python Long Short Term Memory(LSTM), Unity., Asynchronous method, Deep Q-Network(DQN), LSTM, TORCS. Asynchronous met

Size: px

Start display at page:

Download ",.,., ( ).,., A, B. A, B,.,...,., Python Long Short Term Memory(LSTM), Unity., Asynchronous method, Deep Q-Network(DQN), LSTM, TORCS. Asynchronous met"

みさきみのしま
5 years ago
Views:

1 2016 Future University Hakodate 2016 System Information Science Practice Group Report AI Project Name AI love Deep Learning TORCS Deep Learning Group Name TORCS Deep Learning /Project No. 14-B /Project Leader Daichi Fukuda /Group Leader Masafumi Takahashi /Group Member Kaede Noto Sora Ito Masafumi Takahashi Masataka Kato Reina Saito Tomoya Minamoto Advisor Takashi Takenouchi Kiyohito Nagano Kengo Terasawa Yasuhiro Katagiri Date of Submission January 18, 2017

2 ,.,., ( ).,., A, B. A, B,.,...,., Python Long Short Term Memory(LSTM), Unity., Asynchronous method, Deep Q-Network(DQN), LSTM, TORCS. Asynchronous method.,,,,, Unity, TORCS - i -

3 Abstract Recently, AI attracts attention because it can imitate humans in various cases. AI is a kind of technology of Machine Learning. We use it to implement some intelligence. These are same intelligence as human s natural learning abilities. Especially, Deep Learning has archived good results in the field of image processing. In this project, our goal is to imitate and surpass human s thoughts with using Machine Learning. After we discussed our goal, we made two groups. These were group A and group B. The members in group A aimed to develop a combination of pitches expectation system. And the members in group B aimed to develop a car agent that can drive cars faster than humans. We belong to group B and use Deep Reinforcement Learning to develop such a car agent. Deep Reinforcement Learning is technique that applies technique of Deep Learning to function approximation of Reinforcement Learning. The problems to learn in good order are to set appropriate network, rewards, and environment. In the first semester, we implemented Long Short Term Memory (LSTM) on Python, rewards, and environment of a racing game on Unity. And we had the car agent learn with using these works. In the second semester, we had the car agent learn with using techniques of Asynchronous method, Deep Q-Network (DQN), and LSTM on TORCS. TORCS is open source car simulator. Finally, Asynchronous method car agent can drive cars faster in an oval track than humans. Keyword Artificial Intelligence, Machine Learning, Deep Learning, Reinforcement Learning, Unity, TORCS - ii -

4 Python Unity Asynchronous method Deep Q-Network Long Short Term Memory (, Python, Asynchronous method ) Python, DQN (Python, LSTM ) Unity, DQN Unity, LSTM iii -

5 4.3.6 Unity, Asynchronous method Asynchronous method Deep Q-Network Long Short Term Memory Unity Python Asynchronous method Deep Q-Network Long Short Term Memory iv -

6 1 1.1,., ImageNet Large Scale Visual Recognition Challenge 2012(ILSVRC2012), 10% [1].,,., Google Google Brain YouTube, [2]., DeepMind AlphaGo, 4 [3]. 1.2,.,,.,,.,,,. 1.3, DeepMind Deep Q- Network(DQN). DQN,. DQN,,. Atari ,, DQN 1.1. Group Report of 2016 SISP Group Number 14-B

1.1: DQN 1.4 DQN 2. 1,. DQN., DQN,,. 2,.,, 1. DQN 4. 4. 1.5, 2.

7 1.1: DQN 1.4 DQN 2. 1,. DQN., DQN,,. 2,.,, 1. DQN , 2. 1,, 2,,. 2,.,,. Group Report of 2016 SISP Group Number 14-B

8 ,. 2.2, DQN,., 1.4. (1). (2) ,. Unity Python, Experience Replay, Fixed Target Q-Network, Long Short Term Memory(LSTM). Group Report of 2016 SISP Group Number 14-B

9 Unity (1), (2),,,. Experience Replay, Fixed Target Q-Network (1), (2) Deep Q-Network. (1),,. Long Short Term Memory(LSTM) (2), 2.3.3,.,,,., OpenAI OpenAI gym,, ,. TORCS Python, Asynchronous Method, Deep Q-Network, Long Short Term Memory(LSTM). OpenAI gym,. Group Report of 2016 SISP Group Number 14-B

10 TORCS (1), (2),,,. Asynchronous Method, Deep Q-Network, Long Short Term Memory (1), (2) Asynchronous Method, Deep Q-Network, Long Short Term Memory. (1),,. OpenAI gmy OpenAI gym,, Python, Unity Python, Unity 2,,.. Python Python-Unity LSTM. Experience Replay. Fixed Target Q-Network. Unity,.. Python,. Group Report of 2016 SISP Group Number 14-B

11 2.4.2 Python, TORCS. Python Unity, 2 3. Python Unity 1,, Unity, Python.,. Asynchronous method : Asynchronous method,.. Deep Q-Network :. : Deep Q-Network. Long Short Term Memory : Long Short Term Memory,. : Long Short Term Memory,. Group Report of 2016 SISP Group Number 14-B

12 3,. 1, 2, Python, 3 Python., TORCS TORCS, 1 2., 3,.,,. 3.1, Python ( ), Unity 2. Python, Unity. Python, Unity.,,., Python ILSVRC2012 AlexNet. Unity,, 6., MessagePack WebSocket, AlexNet Unity,., Python, Experience Replay LSTM Fixed Target Q-Network 3, Unity,., 3.1,. 3.1: Group Report of 2016 SISP Group Number 14-B

13 3.2, ,.,., gym-torcs TORCS, Unity TORCS.,,,,., 3, 1 2, 3.,, 2 1 3,.,,. Group Report of 2016 SISP Group Number 14-B

14 4, 3., , Python Experimence Replay, Fixed Target Q-Network, Long Short Term Memory,,. Python Unity, Life in Silico, Unity Unity, Unity Unity, C#.,, Unity, Blender 3D,., Python Unity Python, Life in Silico. Group Report of 2016 SISP Group Number 14-B

15 4.2, Asynchronous method Asynchronous method, TORCS, TORCS,., Asynchronous method, Asynchronous method., Asynchronous method. 5.1.,,, OpenAI gym CartPole.,,., CartPole.,., Asynchronous method, Asynchronous method, 2,., 2,,., Asynchronous method, CartPole.,.,.,,,,.,,,.,. Group Report of 2016 SISP Group Number 14-B

16 4.2.2 Deep Q-Network Deep Q-Network, TORCS,., Deep Q-Network., 1 ATARI.,,., Deep Q-Network, OpenAI gym CartPole., TORCS,. CartPole. CartPole, TORCS, TORCS., Deep Q-Network, TORCS,, Long Short Term Memory Long Short Term Memory, CartPole, TORCS.,, LSTM. CartPole, (Cart) (Pole). CartPole,,.,,,.,,,., TORCS, TORCS.,.,,,., TORCS,. TORCS,,. Group Report of 2016 SISP Group Number 14-B

17 4.3, (, Python, Asynchronous method ) (1),,,. (2) Python Unity MessagePack, WebSocket,. (3), Python,,,. (4), OpenAI gym CartPole, 2, 3. (5) Asynchronous method,,. (6),,,.,. Group Report of 2016 SISP Group Number 14-B

18 4.3.2 Python, DQN (1), Python, Unity, Python Unity, Life in Silico( LIS ),,. (2) LIS,,,.. (3) Python 1 1, Fixed Target Q- network,. (4), CartPole Deep Q-Network Experience Replay Fixed Target Q-network. (5) DQN,. (6) CartPole, (Python, LSTM ) (1). (2) Chainer,. (3) Experience Replay,,. (4) gym-torcs Ubuntu. (5) RNN, LSTM. (6) RNN, LSTM, CartPole, TORCS. (7),,,,. Group Report of 2016 SISP Group Number 14-B

19 4.3.4 Unity, DQN (1) Unity, Unity [5]. (2). (3), Slack.,. (4) Blender,,. (5). (6),. (7) gym-torcs Ubuntu. (8), DQN. (9),. (10),,, DQN Unity, LSTM (1),.,. (2), 4. (3) RNN, LSTM,. (4),,. Group Report of 2016 SISP Group Number 14-B

20 4.3.6 Unity, Asynchronous method (1) Unity,,. (2), WebSocket Python, 6. (3),. (4) Python Unity Unity WebSocket. (5),. (6). Group Report of 2016 SISP Group Number 14-B

21 5,. 5.1 Asynchronous method,,.,,.,,., θ θ., (1)θ θ, (2)θ θ dθ, (3)dθ θ,.,, RMSprop 2,. 5.2 Deep Q-Network Deep Q-Network Experience Replay, Fixed Target Q-Network, Clipping 3. Experience Replay,,,,.,.,.. Fixed Target Q-Network, target. TD target θ, θ., target θ, θ. Clipping,, 1, -1.,. Group Report of 2016 SISP Group Number 14-B

22 5.3 Long Short Term Memory Long Short Term Memory. 3. (1),, (2), (3).,,., LSTM,. Group Report of 2016 SISP Group Number 14-B

23 6 B, :, Unity 3D Blender 3D GitHub, Python C# Unity Chainer Group Report of 2016 SISP Group Number 14-B

24 7, Unity. Unity C#. C# II. II, Java., C#.,, Python.,,. I II.,,.,. Group Report of 2016 SISP Group Number 14-B

25 TORCS,., Asynchronous method,.,, Unity, Python Unity Unity. Unity.,. Blender Unity (400m ). 8.1 Unity, 8.2 Blender.,. Python. Unity,. Unity Python. Python. Group Report of 2016 SISP Group Number 14-B

26 8.1: Unity 8.2: Python Python.,. Unity.. LSTM. Group Report of 2016 SISP Group Number 14-B

27 8.3,, 3, Asynchronous method Asynchronous one step Q-Learning, [6]. TORCS. Asynchronous method, TORCS. CartPole Asynchronous method. OU process Deep Q-Network TORCS. GPU CPU. Excel. TORCS Experience Replay. TORCS Fixed Target Q-Network Long Short Term Memory TORCS,,. LSTM, TORCS LSTM.,.,. Group Report of 2016 SISP Group Number 14-B

28 8.4, , : , ,. Group Report of 2016 SISP Group Number 14-B

29 , : ,,..... Group Report of 2016 SISP Group Number 14-B

30 ,.,,.,,. 8.5 B,,, , : 4,. 3. 3,. 4.,. 3. Group Report of 2016 SISP Group Number 14-B

31 8.5.2, : 4, ,,., 5. 5 AI TORCS, ( ).,,,.,,, 5. 5,,.,,,.,,.,, 5. 4, 4. Group Report of 2016 SISP Group Number 14-B

32 8.6, : Python Unity,,.,. Unity,. Unity. Unity..,.,.,,.,. Group Report of 2016 SISP Group Number 14-B

33 :,.,. Deep Q-Network, TORCS, TORCS.,,.,,.,,, AI,.,,.,.,. CartPole DQN TORCS.,,. Group Report of 2016 SISP Group Number 14-B

34 9 9.1, Python, Unity,.,, (Python, ) Python-Unity,, Python LSTM DQN Clipping.,,., LSTM,,. (Python ) LIS,, Python DQN Fixed Target Q-Network., LIS., Fixed Target Q-Network,. (Python ) Python DQN Experience Replay.,,. (Unity ), 4., Group Report of 2016 SISP Group Number 14-B

35 4.,. (Unity ) Unity,, Python.,,. (Unity ), Unity.,,,,. 9.2, , TORCS. Asynchronous method, TORCS., 2.1. Group Report of 2016 SISP Group Number 14-B

36 (Asynchronous method, ) PC GPU, Asynchronous method,,.,. (Deep Q-Network ) CartPole Deep Q-Network Experience Replay, Fixed Target Q-network Clipping., TORCS Experience Replay Fixed Target Q- network., CartPole, TORCS. (Long Short Term Memory ) RNN, LSTM,. TORCS,,.. (Long Short Term Memory ), LSTM. LSTM,,,,,. (Asynchronous method ),,.,. (Deep Q-Network ) Ubuntu TORCS. Group Report of 2016 SISP Group Number 14-B

37 9.2.3,.,.,,, 3.,, TORCS. TORCS.,,..,.,,., TORCS,.,..,,,.,. Group Report of 2016 SISP Group Number 14-B

38 [1] All results, ) [2] Le Q, Ranzato M, Monga R, Devin M, Chen K, Corrado G, Dean J, and Ng A, Building high-level features using large scale unsupervised learning, In ICML, [3] nikkei BP net, , AI ) [4],,, [5], Unity5 3D/2D,, [6] Volodymyr Mnih, Adri Puigdomnech Badia, Mehdi Mirza, Alex Graves, Timothy P Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu, Asynchronous Methods for Deep Reinforcement Learning, In ICML, [7] Simon O Haykin, Neural Networks and Learning Machines, Pearson, [8] Yann LeCun, Leon Bottou, Genevieve B Orr, Klaus Robert Mller, Efficient BackProp, Springer Berlin Heidelberg, [9] Richard S Sutton, Andrew G Barto,,,,, [10] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin Riedmiller, Playing Atari with Deep Reinforcement Learning, NIPS Deep Learning Workshop 2013, [11] Daniele Loiacono, Luigi Cardamone, Pier Luca Lanzi, Simulated Car Racing Championship: Competition Software Manual, [12] Ćirović Velimir, Braking torque control using recurrent neural networks, In Proceedings of the Institution of Mechanical Engineers Part D Journal of Automobile Engineering 226(6), May [13] Sepp Hochreiter, Jrgen Schmidhuber, Long short-term memory, In NEURAL COMPU- TATION Group Report of 2016 SISP Group Number 14-B

,.,., ( ).,., A, B., A, B,.,, 2010 [2], [3],.,,,.,,.,, Python Python 2.., (SVM).,,.,,.,.,.,.,,, - i -

,.,., ( ).,., A, B., A, B,.,, 2010 [2], [3],.,,,.,,.,, Python Python 2.., (SVM).,,.,,.,.,.,.,,, - i - 2016 Future University Hakodate 2016 System Information Science Practice Group Report AI Project Name AI Love Deep Learning AI Group Name AI Scope /Project No. 14-A /Project Leader 1014041 Daichi Fukuda