In response to the challenges of low detection accuracy, missed detections, and false detections in identifying valuable marine organisms in marine ranch areas, this study introduces an enhanced algorithm for detecting rare underwater marine species using the YOLOv8 model. Firstly, a new residual attention mechanism was designed and integrated into the backbone network of the YOLOv8 model to improve focus on the detailed features of underwater targets during feature extraction. Next, a bidirectional feature pyramid with adaptive feature fusion and feature selection characteristics is incorporated into the neck network to effectively combine the strong semantic information of deep feature maps with the localization information of shallow feature maps. This emphasizes the distinctions between the target and the surroundings. The experiment showed that the mean Average Precision (mAP@0.5) of the enhanced YOLOv8 model was 92.98%, which is 1.36 percentage point higher than the original YOLOv8 model. Additionally, the mean Average Precision (mAP@0.5∶0.95) was 76.71%, indicating a 3.7 percentage point improvement over the original YOLOv8 model. Compared with mainstream object detection models such as Faster RCNN, SSD, RetinaNet, YOLOv6, and YOLOv7, the improved model has shown an increase of 1.57 percentage point, 1.74 percentage point, 3.17 percentage point, 4.68 percentage point, and 1.47 percentage point respectively in mAP@0.5. The model proposed in this paper demonstrates high detection accuracy and robust stability in complex seabed environments. It can provide technical support for the scientific management of underwater resources in marine ranches.