Minimizing flight time while ensuring stability and efficiency presents a significant challenge in UAV control. This paper introduces velocity-level and actuator-level control approaches for addressing the minimum-time flight problem of fixed-wing UAVs, utilizing a six degrees of freedom (6-DOF) high-fidelity dynamic model and reinforcement learning. We evaluate four state-of-the-art reinforcement learning algorithms through extensive simulations and compare their performance against traditional methods. The results demonstrate that the proposed velocity-level and actuator-level control methods, based on Proximal Policy Optimization (PPO), achieve a substantial improvement in flight efficiency while maintaining strong generalization and stability. Specifically, the PPO-based actuator-level controller fully leverages the UAV’s maneuverability, resulting in a 27.6% reduction in average flight time compared to traditional controllers. The code is available as open source at https://github.com/running-mars/OpenFlight.