Detecting grape ripeness to synchronously locate picking point during harvesting

Fruit harvesting is often required to synchronously detect ripe grapes and their picking points in robotics. However, conventional asynchronous grape detection cannot fully meet the large-scale production in recent years, such as complex operation, low efficiency, high model redundancy, and high computational cost. In this study, a synchronous model, GrapickNet, was proposed to detect grape ripeness and picking points. An improved Grapick-YOLO12n model was also employed to simultaneously detect grape ripeness and stem points. A bunch-pick point matching algorithm was then utilized to effectively match ripe bunches with their pick points, according to multiple constraint rules (RBPM). 1) Taking YOLO12n-pose as the baseline model, Grapick-YOLO12n framework was integrated the Edge Information Transfer (EIFT) and the C3K2_RFAConv module. Among them, the EIFT module was used to enhance the perception of object boundaries and local elongated small targets using shallow multi-scale edge feature extraction and cross-channel fusion mechanisms. The C3K2_RFAConv module was used to expand the receptive field and strengthen global context modeling for the feature discrimination under complex scenes and multiple targets. 2) Multiple constraints were constructed to improve the successful matching rate between ripe bunches and its picking point using RBPM algorithm. A dataset was constructed as model training data. 4 383 RGB images of greenhouse grapes were captured under two environments, four lighting conditions, and five varieties. Experimental results showed that GrapickNet achieved 88.9% mAP@0.5 for ripeness detection and 82.7% precision for picking point detection. Compared with the baseline model (87.0% mAP@0.5 for ripeness and 77.8% precision for picking points), the performance improved by 1.9% and 4.9%, respectively. Furthermore, the computational complexity of 8.4 GFLOPs and a parameter size of only 3.33 M were obtained with an inference speed of 94.6 FPS. In terms of positioning accuracy, the average errors in the X and Y directions were 2.51 and 5.37 pixels, respectively. The average error was 6.39 pixels in Euclidean distance. The performance of object detection outperformed the existing models, including YOLOv5n-pose, YOLOv8n-pose, YOLOv10n-pose, YOLO11n-pose, YOLO12n-pose, Faster R-CNN, SSD, and RT-DETR. Compared with asynchronous separate detection tasks, GrapickNet shared multi-task synchronous detection for the synchronous, accurate, and efficient detection of fruit ripeness and picking points for different grape varieties in complex greenhouse scenarios. The high accuracy and short detection time were reduced parameter usage for the computational efficiency and deployment on mobile devices. This finding can also provide an efficient and accurate technical solution for grape harvesting robots in semi-structured greenhouses.