Multiagent Deep Reinforcement Learning Algorithms in StarCraft II: A Review

Abstract

StarCraft II, as a real-time strategy game, features multiagent collaboration, complex decision-making processes, partially observable environments, and long-term credit assignment; thus, it is an ideal platform for exploring, validating, and optimizing deep reinforcement learning algorithms. However, designing deep reinforcement learning algorithms that can rival top human players remains a considerable challenge. To assist researchers interested in this field in quickly grasping fundamental concepts and keeping up with the latest research, first, the fundamental theories in the field of reinforcement learning are elucidated, including Markov decision process, Bellman equation, and stochastic game theory. Second, the StarCraft II reinforcement learning platform covering observation space, action space, and reward functions is introduced, and the concepts of macromanagement and micromanagement and their corresponding game maps in StarCraft II are explained. Third, the frameworks and pseudocode of nine mainstream reinforcement learning algorithms designed to achieve macromanagement or micromanagement in StarCraft II are analyzed. Fourth, the micromanagement performance of four mainstream reinforcement learning algorithms is comprehensively evaluated across several types and difficulties of game maps using six metrics, thereby providing a scientific basis for algorithm selection in different application scenarios. Fifth, the challenges faced in developing deep reinforcement learning algorithms for StarCraft II are summarized. This study offers researchers a quick-start guide to StarCraft II reinforcement learning algorithms and facilitates the advancement of theories and methods in the multiagent deep reinforcement learning field, thereby laying the foundation for addressing other real-world complex problems.

Keywords

• Multiagent learning
• reinforcement learning
• StarCraft II
• macromanagement and micromanagement