Tomato is one of the favorite vegetables that is widely grown all over the world. Its harvesting is a typical labor-intensive task. Mechanical harvesting can be expected to promote agricultural development with labor-saving. However, the harvesting robots are often operated under complex greenhouse environments. Among them, an accurate three-dimensional (3D) pose information of the target fruits is required for the selective harvesting. There is a direct impact on the success rate of grasping and picking. In this study, a real-time estimation was proposed for the 3D pose of the tomato fruits using instance segmentation and spatial analysis. The 3D pose information of the target fruits was captured under such challenging conditions. A cascaded network architecture was constructed to combine a lightweight instance segmentation with an improved keypoint detection. A spatial parsing module was integrated with the point cloud in order to achieve high-precision and real-time pose estimation. Firstly, a lightweight instance segmentation model, YOLOv7-M1, was developed to improve the original YOLOv7-seg framework. The CBS backbone of the YOLOv7-seg was reconstructed using the MobileOne module. The computational cost was substantially reduced to maintain the high accuracy of the mask segmentation. The instance segmentation generated the pixel-level fruit masks and regions of interest (ROIs), thereby providing both the keypoint ROIs and prior knowledge of the fruit geometry and surrounding environment. Secondly, the keypoints within the ROIs were detected using an enhanced HRNet model, denoted HRNet-ECA. An Efficient Channel Attention (ECA) mechanism was embedded in this network after the parallel output branches at all four stages of the HRNet. The channel-wise feature selection was improved for the robustness of the keypoint localization in the clustered fruits, occlusions, and non-uniform illumination. A multi-modal framework of the data fusion was constructed to combine the depth map with the ROIs. The point clouds of the target fruits were generated and then processed via color filtering, outlier removal, down-sampling, and least-squares sphere fitting. Real-time geometric computation on the filtered point clouds also yielded the 3D pose of each tomato, including the spatial position and orientation. Experimental results showed that the lightweight instance segmentation successfully improved the real-time performance without compromising the accuracy. Compared with the classical instance segmentation, such as Mask-RCNN and Solov2 models, the improved YOLOv7-M1 model achieved better performance: Average precision increased by 14.35 and 15.21 percentage points, precision by 14.06 and 14.47 percentage points, recall by 13.25 and 11.10 percentage points, and mean Average Precision (mAP₅₀) by 14.30 and 14.51 percentage points, respectively. Furthermore, the GFLOPs of the YOLOv7-M1 were reduced by 30.23%, 54.75%, and 29.99% , while the frame rate increased by 10.45%, 14.16%, and 8.34%, respectively, compared with the YOLOv7-seg, YOLOv8l-seg, and YOLOv11l-seg. In the keypoint detection stage, three attention mechanisms—SE, CBAM, and ECA—were embedded into the HRNet for comparison. The similarity of the keypoint was improved by 0.87, 1.68, and 2.23 percentage points, respectively. The ECA provided the largest accuracy, while the GFLOPs increased by less than 1%, thus meeting real-time constraints. The overall framework of the pose estimation was further validated on 100 groups of tomato data. The average accuracy of the pose estimation reached 95.00%, the mean 3D orientation error was 9.40°, and the mean error of the keypoint localization was 4.13 mm. The average position errors in the X, Y, and Z directions were 3.41, 2.95, and 1.02 mm, respectively. The average processing time per fruit was 0.063 s. Finally, the improved model was deployed on a tomato harvesting robot and then tested in a real greenhouse. Among 50 harvesting trials, 44 were successful, corresponding to a success rate of 88.00%. These findings can provide reliable information on the fruit pose to guide the end effector. A favorable balance between accuracy and real-time performance can offer an efficient solution for the precise and mechanical harvesting in complex agricultural environments.

Selective picking of ripe cucumber fruits can often be realized under unstructured environments, where the cucumbers grow. However, manual harvesting is high cost and labor intensity. Cucumber-picking robots can be expected to reduce the manpower requirements in modern agriculture. Therefore, the vision system can dominate the accurate and rapid recognition of the fruit, leading to the high efficiency of the robot picking. This study aims to achieve the efficient selective picking of cucumber fruits under complex environments, such as light changes. The RT-Detr-EV model was also proposed to take the RT-Detr as the baseline network. The cucumber was selected as the research object. Firstly, the Re-parameterization VGG module was added into the backbone network. A multi-branch structure was then adopted during training, in order to strengthen the feature extraction of the network for the high recognition accuracy. While the multi-branch structure was merged during inference. The complexity of the network and the amount of computation were reduced to optimize the inference performance; Secondly, the lightweight cascade in the neck network was added into the grouping self-attention mechanism module, in order to reduce the computational overhead. Thus the high detection speed of the model was obtained to increase the depth of the network; Finally, Minimum Point Distance based Intersection Over Union (MPDIoU) also replaced the loss function in the original model. All the intersection and merger ratios were considered to calculate the regression loss of the target frame. The convergence of the model was accelerated to improve the detection accuracy. The results show that the mean average detection precision and speed of the improved RT-Detr-EV reached 95.8% and 61.3 frames/s, respectively, compared with the original model by 3.2 percentage points and 17.4 frames/s, respectively. The accuracy of identifying cucumbers that are not suitable for picking has increased by 4.6 and 6.5 percentage points, respectively, compared with the YOLOv7-X and YOLOv8-l. While the detection speed was improved by 40.6 and 25frames/s, respectively. The number of parameters was reduced by 55.5% and 27.3%, respectively. Meanwhile, the multi-scene application tests show that the RT-Detr-EV shared the high detection accuracy under different light angles. In the smooth light environment, the improved model was improved by 1.2 and 2.8 percentage points, compared with the YOLOv8 and RT-Detr model, respectively; There was improved by 4.5 and 5.5 percentage points under the backlight environment, respectively. Once the exposure level of the picking scene was in the interval of 40%-160%, the mAP₅₀ variations in the improved model were less than or equal to 0.2 and 0.5 percentage points in the smooth light and the backlight condition, respectively; The original magnitude was much smaller than that of the YOLOv8 and RT-Detr under various picking scenes with different light angles. Therefore, the improved model shared the better robustness and generalization udner complex picking scenarios with multiple changes in lighting conditions. In conclusion, the RT-Detr-EV network model was verified with the better performance index in the target detection task during fruit picking under the complex growth environment. The finding can also provide a valuable reference for the target localization task in the selective picking robots.