SC-MAIRL: Semi-Centralized Multi-Agent Imitation Reinforcement Learning

Abstract

Multi-agent reinforcement learning (MARL) is a challenging branch of reinforcement learning that requires cooperation of interactive learning agents to achieve individual objectives as well as shared team objectives. Existing MARL algorithms generally use either centralized global state representation or decentralized local observation to perform training and execution. In this paper, we introduce a novel MARL learning paradigm, centralized training with semi-centralized execution (CTSCE), and present a new MARL algorithm for addressing multi-agent problems: Semi-Centralized Multi-Agent Imitation Reinforcement Learning (SC-MAIRL). The semi-centralized approach aggregated with agents’ spatial and temporal information serves as a joint knowledge base to facilitate a learning agent to discover team objectives and make fine-grained decisions. We also utilize a pre-trained performant teacher policy to guide an untrained model towards positive game states as a form of imitation learning, significantly increasing the agent’s learning speed. In addition, to encourage agents to learn both offensive and defensive behaviors and smooth the high-dimensional learning curve, we present a new set of reward-shaping functions to further improve SC-MAIRL’s learning performance. Our approach is evaluated using one of the most challenging scenarios within the StarCraft Multi-Agent Challenge environment, and the results show that SC-MAIRL outperforms the state-of-the-art MARL algorithm MAPPO in several metrics and allows our agents to learn and employ novel, complex macro strategies more effectively.

Keywords

• CTSCE
• influence maps
• multi-agent systems
• semi-centralized reinforcement learning
• SMAC
• StarCraft II