Visible and infrared (RGB-IR) fusion object detection plays an important role in security, disaster relief, etc. In recent years, deep-learning-based RGB-IR fusion detection methods have been developing rapidly, but still struggle to deal with the complex and changing scenarios captured by drones, mainly due to two reasons: (A) RGB-IR fusion detectors are susceptible to inferior inputs that degrade performance and stability. (B) RGB-IR fusion detectors are susceptible to redundant features that reduce accuracy and efficiency. In this paper, an innovative RGB-IR fusion detection framework based on global–local feature optimization, named GLFDet, is proposed to improve the detection performance and efficiency of drone-captured objects. The key components of GLFDet include a Global Feature Optimization (GFO) module, a Local Feature Optimization (LFO) module and a Channel Separation Fusion (CSF) module. Specifically, GFO calculates the information content of the input image from the frequency domain and optimizes the features holistically. Then, LFO dynamically selects high-value features and filters out low-value features before fusion, which significantly improves the efficiency of fusion. Finally, CSF fuses the RGB and IR features across the corresponding channels, which avoids the rearrangement of the channel relationships and enhances the model stability. Extensive experimental results show that the proposed method achieves the best performance on three popular RGB-IR datasets DroneVehicle, VEDAI, and LLVIP. In addition, GLFDet is more lightweight than other comparable models, making it more appealing to edge devices such as drones. The code is available at https://github.com/laochen330/GLFDet.