Policy training against diverse opponents remains a challenge when using Multi-Agent Reinforcement Learning (MARL) in multiple Unmanned Combat Aerial Vehicle (UCAV) air combat scenarios. In view of this, this paper proposes a novel Dominant and Non-dominant strategy sample selection (DoNot) mechanism and a Local Observation Enhanced Multi-Agent Proximal Policy Optimization (LOE-MAPPO) algorithm to train the multi-UCAV air combat policy and improve its generalization. Specifically, the LOE-MAPPO algorithm adopts a mixed state that concatenates the global state and individual agent’s local observation to enable efficient value function learning in multi-UCAV air combat. The DoNot mechanism classifies opponents into dominant or non-dominant strategy opponents, and samples from easier to more challenging opponents to form an adaptive training curriculum. Empirical results demonstrate that the proposed LOE-MAPPO algorithm outperforms baseline MARL algorithms in multi-UCAV air combat scenarios, and the DoNot mechanism leads to stronger policy generalization when facing diverse opponents. The results pave the way for the fast generation of cooperative strategies for air combat agents with MARL algorithms.
- Article type
- Year
- Co-author
Open Access
Full Length Article
Issue
In order to solve the problem of time cooperative path planning for unmanned marine system (UMS) in complex environments, we propose a cooperative path planning method based on a differential evolution (DE) algorithm.
First, natural environment and enemy threat models in the air, sea and underwater are established. The objective function of path planning with multiple constraints is constructed and a strategy proposed to solve the time coordination problem under the constraints of heterogeneous platforms starting at different times and places, but arriving at the same time and place. Next, a DE algorithm is used to optimize the solution, and the influence of the number of waypoints on the success rate of the planning is studied.
Based on the reasonable selection of the number of waypoints, the DE algorithm can respectively plan the path that satisfies the time cooperative constraint for the UMS and find the shortest time to reaching the target in complex environments with multiple threats and obstacles. The results also show that either excessive or inadequate numbers of waypoints will result in success rate reduction of the path planning.
The results of this study show that the model and constraint conditions are reasonable, and the proposed algorithm can solve the problem of time coordination in the path planning of UMS, which has great practical application value.
京公网安备11010802044758号