Lightweight instance segmentation method for the key parts of chicken carcass based on improved YOLOv12n-seg

Accurate segmentation of key parts in the chicken carcass is crucial for intelligent cutting systems in the modern poultry processing industry. However, the false detection, missed detection, and inaccurate segmentation can occur in key parts of chicken carcasses under complex industrial scenarios (e.g., adhesion between wings and drumsticks, occlusion, and uneven lighting). This study aimed to develop a lightweight, high-precision, and real-time instance segmentation model suitable for deployment on intelligent chicken carcass cutting equipment. An enhanced dataset was constructed for chicken carcasses, with emphasis on Sanhuang and white-feathered chickens. Furthermore, the 109 0 original images were expanded into 545 0 images, thus covering three types of scenarios: ambient lighting, carcass occlusion, and compression-induced deformation. Multi-dimensional data augmentation techniques such as geometric transformation, illumination adjustment, and occlusion simulation were adopted to improve the model’s robustness. DEF-YOLO-seg model was developed to improve the YOLOv12n-seg as the baseline: (1) C3k2_DAttention module was designed to fuse the C3k2 module with Deformable Attention (DAttention), which replaced the Area-Attention Enhanced Cross-Feature (A2C2f) module in the lower layer of the backbone network for the feature extraction in the adhered/occluded regions; (2) Efficient Up-Convolution Block (EUCB) was introduced to replace the Upsample module in the neck network, thus reducing computational cost for the feature fusion efficiency; (3) A composite loss function (Focaler-CIoU) with Focaler-IoU and CIoU was constructed for the distribution of easy and difficult samples under complex scenarios. Finally, model training and testing were completed on a hardware platform with an NVIDIA RTX 3090 GPU and an Intel Xeon Platinum 8362 CPU. The DEF-YOLO-seg model achieved a mean Average Precision at an IoU threshold of 0.5 (mAP₅₀) of 95.5% and a mean Average Precision at IoU thresholds from 0.5 to 0.95 (mAP_50-95) of 94.1%, which were 1.3 and 2.8 percentage points higher than those of the baseline YOLOv12n-seg, respectively. With a parameter count of 3.3M and a computational complexity of 11GFLOPs, the model’s inference time per image on a local computer was no more than 30 ms. Compared with mainstream models, such as YOLOv9c-seg, YOLOv11n-seg, and YOLOv12n-seg, the improved model maintained lightweight for the superior segmentation accuracy. Furthermore, the parameter sensitivity analysis revealed that the optimal Focaler-CIoU configuration (d=0.22, u=0.73) precisely matched the IoU distribution of chicken carcass data. The task-specific loss function was designed for rather than generic settings. There was the an image-level accuracy of 95.0%. Dice coefficients of the neck, wings, and drumsticks increased from 0.85, 0.83 and 0.78 to 0.93, 0.92 and 0.90, respectively, in the practical production line. The improved model was effectively reduced the missed detection, false detection, and false segmentation of small parts (e.g., neck and shank) under adhesion and occlusion. The DEF-YOLO-seg model also achieved the a better balance among segmentation accuracy, real-time performance, and deployment feasibility in intelligent cutting equipment for chicken carcasses. The findings can provide the technical support to the intelligent upgrading for theof food processing. Future research can focus on cutting path planning, as well as the balance between lightweight and detection accuracy.