Distributed UAV formation control with virtual structure guided reinforcement learning

In single decision-making models based on reinforcement learning algorithms, the adaptability is often insufficient when handling complex Unmanned Aerial Vehicle (UAV) formation tasks due to limited autonomous decision-making capabilities. To address this, this paper proposes a distributed decision-making method guided by the virtual structure approach integrated with a deep reinforcement learning algorithm. First, to reduce the difficulty of strategy optimization for reinforcement learning algorithms in diverse task environments, the overall task is functionally decomposed. Local task planning is then implemented for individual task scenarios, such as static obstacles, random obstacles, and communication interference. Multiple decision sub-models are constructed along with the design of the calling process between these models. Next, to enhance guidance, the virtual structure method is integrated with the Soft Actor-Critic (SAC) reinforcement learning algorithm to build a distributed decision-making framework. Through decentralized training of each sub-model, the success rate and flexibility of task execution are significantly improved. Finally, a centralized execution approach is adopted, where environmental changes serve as the triggering condition for the dynamic selection and seamless switching betweeen sub-models. This allows the UAV formation to autonomously adjust its formation according to changes in the task environment, achieving the mission objectives while significantly enhancing the overall adaptability and survivability of the swarm. The effectiveness of the method is validated through simulation experiments in multiple scenarios.