Continuous Action Air Combat Maneuver Decision-Making Based on T-MGMM

Abstract

In autonomous air combat, tactics are inherently complex, and control inputs are continuous. Traditional reinforcement learning (RL) algorithms often rely on discretization or independent Gaussian assumptions, which fail to capture correlations between control variables, limiting the expressiveness of strategies. Moreover, the highly dynamic and complex nature of battlefield scenarios poses significant challenges for conventional neural networks in modeling the long-term evolution of sequential data. To address these challenges, this paper proposes a novel algorithm, T-MGMM, which integrates Transformer networks with a Multivariate Gaussian Mixture Model (MGMM). The self-attention mechanism of Transformers effectively captures dependencies between variables and key situational information. Meanwhile, MGMM utilizes non-diagonal covariance matrices to account for correlations between actions, enhancing action modeling. This synergy ensures precise sequence modeling and flexible decision-making, making T-MGMM particularly well-suited for the complexities of air combat scenarios. To further improve optimization stability, we introduce internal Kullback-Leibler divergence regularization. Experimental results demonstrate that T-MGMM outperforms state-of-the-art algorithms, achieving higher Elo scores within the same training steps, and showcasing superior effectiveness and robustness in air combat decision-making.

Keywords

• Air combat
• deep reinforcement learning
• maneuver decision-making
• continuous action space
• Transformer
• Gaussian mixture model