Amid the rapid expansion of the low-altitude economy, UAVs present escalating security threats to large-scale public spaces. Accurate and rapid target detection of drones has become the primary and critical component in anti-drone systems. To address the critical challenges of low detection accuracy, high miss rate, and false positives in small UAV detection within complex environments, this study proposes YOLOv11-drone, an optimized object detection framework based on an enhanced YOLOv11 architecture specifically designed for counter-UAV applications. First, an autonomous dataset of small-target drones in complex backgrounds was constructed. Second, to improve the model’s ability to extract features for drone targets, a partial spatial-channel cooperative attention module C2SCSPSA was created that combines channel attention with spatial attention processes. Furthermore, a small-target-oriented multi-level feature pyramid network (STFPN) was proposed. By fully leveraging shallow semantic information, this architecture significantly reduces the model’s parameter count while improving its focus on small targets. Finally, the model’s loss function was replaced with the EIoU metric, which simultaneously enhances regression accuracy and accelerates convergence speed. The proposed improved algorithm was evaluated on the self-constructed dataset. Experimental results demonstrate that the YOLOv11-drone model achieves a detection accuracy of 95.5%, representing a 4.1% improvement over the baseline algorithm. Additionally, the model reduces parameter count by 62% while attaining an inference speed of 83 frames per second. These advancements confirm its efficacy for small-target drone detection in complex backgrounds. To further evaluate generalization capability, experiments were conducted on the public VisDrone2019 benchmark. The suggested technique shows strong cross-domain applicability and generalization performance, achieving a 0.6% improvement in mAP@0.5 while using just 38% of the baseline model’s parameters.