Rapid and accurate detection and classification of bridge defects are fundamental to maintaining bridge safety. However, the complex environment and diverse defect shapes pose significant challenges to detection algorithms. To address these challenges, this paper proposes a high-performance bridge object detection model named GDS-YOLO. Firstly, a GDS-Neck structure based on the Gather-and-Distribute mechanism(GD) and Scale Sequence Feature Fusion module (SSFF) is designed for and applied to enhance the feature fusion capabilities of the neck part of the model. Furthermore, a lightweight detection head called P-Head, which utilizes Partial Convolution (PConv), is developed to reduce the computational complexity and improve detection speed. In addition, the SimAM attention mechanism is introduced to the backbone of the model, further enhancing the model's feature extraction ability. The experimental analysis based on the open-source bridge defect dataset and field-collected data reveals that compared to You Only Look Once version 8(YOLOv8), The Giga Floating Point Operations per Second (GFLOPs) of GDS-YOLO has been reduced by 0.3, with a 3.5% improvement in  and a 3.9% enhancement in . Therefore, the GDS-YOLO algorithm exhibits lower computational requirements and superior performance in complex bridge detection environments, ensuring accurate detection and classification of various defect types. This enhancement increases the safety and stability of bridge defect detection, providing theoretical research and technical support for bridge defect inspection and maintenance.