Fabric defect detection plays a vital role in ensuring textile quality. However, traditional manual inspection methods are often inefficient and inaccurate. To overcome these limitations, we propose FD-YOLO, an enhanced lightweight detection model based on the YOLOv11n framework. The proposed model introduces the Bi-level Routing Attention (BRAttention) mechanism to enhance defect feature extraction, enabling more detailed feature representation. It proposes Deep Progressive Cross-Scale Fusion Neck (DPCSFNeck) to better capture small-scale defects and incorporates a Multi-Scale Dilated Residual (MSDR) module to strengthen multi-scale feature representation. Furthermore, a Shared Detail-Enhanced Lightweight Head (SDELHead) is employed to reduce the risk of gradient explosion during training. Experimental results demonstrate that FD-YOLO achieves superior detection accuracy and Lightweight performance compared to the baseline YOLOv11n.