A sample selection mechanism for multi-UCAV air combat policy training using multi-agent reinforcement learning

Policy training against diverse opponents remains a challenge when using Multi-Agent Reinforcement Learning (MARL) in multiple Unmanned Combat Aerial Vehicle (UCAV) air combat scenarios. In view of this, this paper proposes a novel Dominant and Non-dominant strategy sample selection (DoNot) mechanism and a Local Observation Enhanced Multi-Agent Proximal Policy Optimization (LOE-MAPPO) algorithm to train the multi-UCAV air combat policy and improve its generalization. Specifically, the LOE-MAPPO algorithm adopts a mixed state that concatenates the global state and individual agent’s local observation to enable efficient value function learning in multi-UCAV air combat. The DoNot mechanism classifies opponents into dominant or non-dominant strategy opponents, and samples from easier to more challenging opponents to form an adaptive training curriculum. Empirical results demonstrate that the proposed LOE-MAPPO algorithm outperforms baseline MARL algorithms in multi-UCAV air combat scenarios, and the DoNot mechanism leads to stronger policy generalization when facing diverse opponents. The results pave the way for the fast generation of cooperative strategies for air combat agents with MARL algorithms.