Trajectory planning of solar powered unmanned aerial vehicles based on multi-objective reinforcement learning

There is a significant coupling between the influencing factors of high-altitude long-endurance solar-powered UAVs in harvesting solar energy and gradient wind energy, and optimizing the harvesting efficiency of these two types of energy simultaneously often leads to conflicts. To address this issue, this study proposes a trajectory planning method based on multi-objective reinforcement learning. This method adopts the multi-objective Soft Actor-Critic (SAC) algorithm based on the multi-objective Markov decision process, combines the UAV's energy harvesting power and energy consumption power into a reward vector, and adds randomly generated weights in each update step. The converged trained policy network can output thrust, attack angle, and bank angle commands based on flight information and a given weight vector, enabling the generation of a set of energy-optimal trajectory solutions within the weight space. Simulation results show that compared with the minimum energy consumption strategy and the strategy based on the conventional single-objective SAC algorithm, this method consistently achieves better energy optimization efficiency and can adaptively respond to weight changes of energy objectives. Compared with the offline optimized trajectory solution set based on the Non-dominated Sorting Genetic Algorithm Ⅱ, this method achieves a hypervolume of the trajectory solution set reaching 90.07% of the former while maintaining excellent real-time performance. In addition, this method also demonstrates a certain degree of generalization ability and can adapt to new untrained wind fields.