An accurate planning of the harvesting priority is often required for the asynchronous maturity in the large-scale facility-grown tomatoes. However, the asynchronously ripening has posed a great challenge on the harvest scheduling and fruit quality. In this study, a harvesting decision-making framework (DeepSeek-VKQ) was proposed for the asynchronously ripened fruits of the facility-grown tomato using DeepSeek-7B. Enhanced visual perception, structured knowledge retrieval, and logical reasoning were synergistically combined in the framework. Firstly, a comprehensive corpus was constructed using the basic knowledge of the tomato cultivation, ripening stage, and pest/disease strategies. Multi-source data was also integrated from the technical manuals, academic literature, web resources, and expert experience. The 7,000 structured Q&A pairs were developed for the knowledge reasoning. Architectural refinements were made on the YOLOv11n backbone network, in order to enhance the visual feature extraction. Global attention mechanism (GAM) was effectively suppressed the foliar background interference. While the conventional CIoU loss was replaced with the spatial SIoU loss, in order to significantly enhance the bounding box regression accuracy under occlusion scenarios. Structured transformation was involved the spatial discretization of the fruit coordinates, and the fusion of the detection confidence with the maturity levels into a continuous ripeness index ranging from 0 to 1.0. Non-linear weight function was used to modulate the probability flow between adjacent maturity levels. Discrete detection outputs were transformed into the continuous index, where the confidence scores were integrated with the maturity labels to indicate the gradual ripening. Effective mapping was realized from the detection to decision semantics. Knowledge reasoning was relied mainly on the dynamic knowledge base. Multi-source textual knowledge was mapped into a low-dimensional semantic space. As such, the feature vectors were generated using the BGE-M3 semantic embedding model. Vector database was also employed to store these vectors. Efficient retrieval was then and facilitated to link the vector indexes into their original knowledge. The key environmental parameters were real-time acquired using API interfaces of meteorological platforms. Precise semantic matching was provided for the chain of thought (CoT) decomposition-guided reasoning. Ultimately, there was the deep integration between large language model (LLM) and dynamically updated agronomic-meteorological knowledge. Experiments were also validated on the annotated tomato images. It was found that the visual extractor was achieved 87.6% mean average precision (mAP) at 0.5-0.95 IoU thresholds, indicating 2.5, 3.2, and 2.9 percentage points over YOLOv12n, YOLOv13n, and RT-DETRv2, respectively, with the inference time of 10.2 ms. The exceptional performance of the framework was achieved in the decision-making tasks during tomato harvesting. The better performance was obtained with the precision, recall, and F1-score of 88.4%, 91.7%, and 90.0%, respectively. Compared with the original DeepSeek-7B model, these metrics were significantly improved by 21.0, 18.0 and 19.6 percentage points, respectively. Ablation experiments showed that there were the F1-score contributions of 7.8 percentage points from the vision module, 6.6 percentage points from the knowledge retrieval, and 3.6 percentage points from the CoT decomposition. Their contributions were accounted for the overall performance. Compared with the benchmarks, the 7B-parameter DeepSeek-VKQ substantially outperformed several larger open-source multimodal models, thereby exceeding GLM-4V-9B, InternLM3 (20B), Qwen2.5-VL (72B), and DeepSeek-VL2 (27.5B) by 16.1, 17.2, 10.8, and 12.6 percentage points in the F1-score, respectively. Notably, the performance even approached that of the leading closed-source multimodal models, with an F1-score of 90.0% trailing the 90.9% of GPT-4o by a marginal 0.9 percentage points, a recall rate of 91.7% surpassing GPT-4o’s 90.0%, and a precision of 88.4% in a narrow gap with GPT-4o’s 91.8%. Importantly, all of these were achieved with a fraction of the parameter scale. Task-specific evaluations were maintained the hallucination rates below 6.5% over all subtasks. Furthermore, the third-party large language model was evaluated the framework performance over diverse tasks. Thereby the reliability was enhanced in the practical applications. Cross-modal perception, knowledge retrieval, and logical reasoning were integrated to enhance the framework for the high precision of the tomato harvest decision-making. The finding can also provide the effective technical support for the robotic harvesting decision-making in the facility-grown tomato cultivation.

This study aimed to address the challenge of accurately and reliably detecting tomatoes in dense planting environments, a critical prerequisite for the automation implementation of robotic harvesting. However, the heavy reliance on extensive manually annotated datasets for training deep learning models still poses significant limitations to their application in real-world agricultural production environments. To overcome these limitations, we employed domain adaptive learning approach combined with the YOLOv5 model to develop a novel tomato detection model called as TDA-YOLO (tomato detection domain adaptation). We designated the normal illumination scenes in dense planting environments as the source domain and utilized various other illumination scenes as the target domain. To construct bridge mechanism between source and target domains, neural preset for color style transfer is introduced to generate a pseudo-dataset, which served to deal with domain discrepancy. Furthermore, this study combines the semi-supervised learning method to enable the model to extract domain-invariant features more fully, and uses knowledge distillation to improve the model's ability to adapt to the target domain. Additionally, for purpose of promoting inference speed and low computational demand, the lightweight FasterNet network was integrated into the YOLOv5's C3 module, creating a modified C3_Faster module. The experimental results demonstrated that the proposed TDA-YOLO model significantly outperformed original YOLOv5s model, achieving a mAP (mean average precision) of 96.80% for tomato detection across diverse scenarios in dense planting environments, increasing by 7.19 percentage points; Compared with the latest YOLOv8 and YOLOv9, it is also 2.17 and 1.19 percentage points higher, respectively. The model's average detection time per image was an impressive 15 milliseconds, with a FLOPs (floating point operations per second) count of 13.8 G. After acceleration processing, the detection accuracy of the TDA-YOLO model on the Jetson Xavier NX development board is 90.95%, the mAP value is 91.35%, and the detection time of each image is 21 ms, which can still meet the requirements of real-time detection of tomatoes in dense planting environment. The experimental results show that the proposed TDA-YOLO model can accurately and quickly detect tomatoes in dense planting environment, and at the same time avoid the use of a large number of annotated data, which provides technical support for the development of automatic harvesting systems for tomatoes and other fruits.

Accurate peach detection is a prerequisite for automated agronomic management, e.g., peach mechanical harvesting. However, due to uneven illumination and ubiquitous occlusion, it is challenging to detect the peaches, especially when the peaches are bagged in orchards. To this end, an accurate multi-class peach detection method was proposed by means of improving YOLOv5s and using multi-modal visual data for mechanical harvesting in this paper. RGB-D dataset with multi-class annotations of naked and bagging peach was proposed, including 4127 multi-modal images of corresponding pixel-aligned color, depth, and infrared images acquired with consumer-level RGB-D camera. Subsequently, an improved lightweight YOLOv5s (small depth) model was put forward by introducing a direction-aware and position-sensitive attention mechanism, which could capture long-range dependencies along one spatial direction and preserve precise positional information along the other spatial direction, helping the networks accurately detect peach targets. Meanwhile, the depthwise separable convolution was employed to reduce the model computation by decomposing the convolution operation into convolution in the depth direction and convolution in the width and height directions, which helped to speed up the training and inference of the network while maintaining accuracy. The comparison experimental results demonstrated that the improved YOLOv5s using multi-modal visual data recorded the detection mAP of 98.6% and 88.9% on the naked and bagging peach with 5.05 M model parameters in complex illumination and severe occlusion environment, increasing by 5.3% and 16.5% than only using RGB images, as well as by 2.8% and 6.2% when compared to YOLOv5s. As compared with other networks in detecting bagging peaches, the improved YOLOv5s performed best in terms of mAP, which was 16.3%, 8.1% and 4.5% higher than YOLOX-Nano, PP-YOLO-Tiny, and EfficientDet-D0, respectively. In addition, the proposed improved YOLOv5s model offered better results in different degrees than other methods in detecting Fuji apple and Hayward kiwifruit, verified the effectiveness on different fruit detection tasks. Further investigation revealed the contribution of each imaging modality, as well as the proposed improvement in YOLOv5s, to favorable detection results of both naked and bagging peaches in natural orchards. Additionally, on the popular mobile hardware platform, it was found out that the improved YOLOv5s model could implement 19 times detection per second with the considered five-channel multi-modal images, offering real-time peach detection. These promising results demonstrated the potential of the improved YOLOv5s and multi-modal visual data with multi-class annotations to achieve visual intelligence of automated fruit harvesting systems.