Seafloor object detection can be expected to increasingly promote the marine economy in a wide range of applications, including ecological monitoring, sustainable aquaculture, and marine resource exploration. However, several challenges have generally remained in detecting the marine benthic targets, such as hard-to-capture edge features, scale variations, and the dense distribution of small targets under seafloor environments. Conventional algorithms of object detection are applied to such complex underwater scenarios. It is still lacking in sufficient adaptability and robustness of the detection. Therefore, it is often required for high accuracy and reliability in practical applications. In this study, an enhanced marine benthic organism detection (MBOD) was proposed for high performance using YOLOv11n, namely YOLOv11n-MBOD. The original YOLOv11n architecture was replaced with the original modules. Firstly, context guide block downsampling (CGDown) was introduced to combine with the improved deformable adaptive fusion downsampling (DAFDown) for the hierarchical cooperative downsampling mechanism. Specifically, rich features were learned from the small targets and objects with irregular shapes, thereby enhancing the overall feature extraction of the network. Secondly, a lightweight module was developed for the feature enhancement, the reparameterized progressive convolution block (ReProBlock). The extraction and fusion of features were optimized from the fine-grained local features to the broader global multi-region information. As such, the network was used to capture subtle textures, edge contours, and important structural information of benthic organisms. In addition, a task-aware interactive head (TAI-Head) was proposed to facilitate the effective information interaction between localization and classification tasks. Complex underwater features were effectively compensated after optimization. Finally, a composite loss function with normalized Wasserstein distance (NWD) and weighted minimum point distance intersection over union (Wise-MPDIoU) was introduced to improve both the accuracy and stability of predicted bounding boxes. Experimental results demonstrated that the precision, recall, and mean average precision (mAP50) of the YOLOv11n-MBOD model increased by 1.5, 2.6, and 2.6 percentage points, respectively, compared with the baseline model (YOLOv11n), indicating the better performance of detection. The number of parameters was reduced by 0.21 M, compared with the baseline, whereas the computational cost slightly increased to 9.6 G. The YOLOv11n-MBOD model significantly reduced the overhead for the small parameter counts and low computational costs, compared with the Faster R-CNN and YOLOv11s. The precision, recall, and mAP50 were 0.6, 2.5; 0.1, 7.8; and 0.3, 6.6 percentage points higher than those of YOLOv11s and Faster R-CNN, respectively. The precision, recall, and mAP50 were higher by 0.8, 0.3, and 0.6 percentage points, respectively, indicating an excellent model of seafloor object detection, compared with the CEH-YOLO. Furthermore, the precision, recall, and mAP50 of YOLOv11n-MBOD were 1.2-3.4, 2.4-4.4, and 2.8-4.3 percentage points higher than those of YOLOv5n, YOLOv8n, YOLOv10n, YOLO12, and YOLO13, respectively, compared with the mainstream models at a similar scale under parameter counts and computational costs. Visualization results showed that the best performance of YOLOv11n-MBOD was achieved in the practical scenarios under complex seafloor environments, such as blurred edges, low-light occlusion, dense small objects, and overlapping objects. There were no missed detections or false positives. The high accuracy, strong robustness, and excellent generalization of YOLOv11n-MBOD were also realized for the flexible adaptability of the improved model to unknown complex seafloor scenarios. The outstanding detection accuracy can be expected to effectively cope with various complex seafloor conditions. The finding can provide robust technical support for the aquaculture operations and marine ecological monitoring.