Development of weed detection and management system using multi-modal information fusion for wheat fields

Weeds can pose a serious threat to the wheat production in recent years, leading to the yield reduction and quality deterioration. Current chemical control has been the primary removal of weeds in wheat fields. However, it is often required to accurately identify and locate the weeds, in order to avoid the excessive pesticide application, low utilization efficiency, and severe environmental pollution. Therefore, there is an urgent need for the rapid and accurate identification of weeds during wheat production. The variable-rate application can also be used to guide the weed distribution mapping and operational decision making. Weed detection in wheat fields is essential to the precision weeding for the high accuracy and efficiency. Existing studies can rely primarily on the manually designed features, such as the spectral, color, texture, and positional information, in order to detect the broadleaf weeds during the wheat seedling stage. However, it is difficult to distinguish the wheat from the grass weeds, due to their similar morphology. Furthermore, the high precision removal of the weeds is required for the accurate detection and the generation of spatial distribution maps. Previous studies on agricultural weeding have focused mainly on the navigation and control system. It is still lacking on the protocols. Navigation systems have relied typically on the crop row features and coordinate information for the path planning, without considering the actual weed distribution in the field. In this study, a detection system was proposed for the weeds in the wheat fields using multimodal fusion. Accurate detection was performed on the various weed species, particularly grass weeds that closely resemble wheat. A dual-branch network was designed to simultaneously extract the features from RGB and depth images. The feature maps from the different convolutional layers were also fused using a multi-scale object detection. Finally, an attention mechanism was employed for the adaptive multimodal feature fusion. Additionally, the spatial distribution and removal protocol of weeds were introduced after mapping. According to the weed detection and the image coordinate, the weed species, area, location, distribution maps were generated for the different weed types. Their spatial distribution was also visualized in wheat fields. A weed index system was proposed for the different species. A ratio-based algorithm was utilized to quantify the weed occurrence, providing a strong reference for the herbicide selection. Considering the operational constraints of the agricultural machinery, the K-means clustering algorithm was applied into the weed regions, in order to determine the spray area and herbicide dosage. The weed protocols were tailored for the precision weeding equipment, thus offering the decision-making and technical assistance. Field tests demonstrate that the multimodal fusion model was significantly improved the detection accuracy of the weeds. Compared with the single-modal RGB images, the detection accuracy increased by 13.1 percentage points for the grass weeds. Performance and functionality tests confirm that the control system operated stably across multiple platforms, thus achieving the real-time and accurate detection of various weed species, together with the decision information. The multimodal fusion can provide the critical technical support to detect the grass weeds for the precision weeding.