Online detection of wheat lodging area from the perspective of harvester based on improved DeepLabv3+

Wheat harvesting robots can frequently confine the key challenges during actual field operations, such as the missed or incomplete harvesting in the lodging areas, and header blockages. In this study, an online detection was developed to monitor the wheat lodging areas using an improved DeepLabv3+ semantic segmentation. The environmental perception and recognition of harvesting robots were enhanced in the complex and variable farmland, thereby supporting highly efficient and low-loss mechanized harvesting. The dataset was constructed with 4 250 high-resolution images under the lodging scenarios in wheat fields using a ZED2i vehicle-mounted binocular RGB camera. Data collection was conducted in June 2023 in the main winter wheat production zone of Xincao Farm, Yancheng, Jiangsu Province, China. Data augmentation was applied to improve the generalization of the model under diverse field conditions, including random color adjustment, image rotation, and contrast enhancement, simulation in ambient lighting, and motion blur induced by machinery vibration. Each image was annotated at the pixel level using the LabelMe tool, allowing for the precise semantic segmentation training and evaluation. Furthermore, a lightweight and high-performance segmentation model, named MVDC-DeepLabv3+, was developed to fully meet the real-time field deployment. Three architectural enhancements were introduced as follows. Firstly, the original Xception backbone of DeepLabv3+ was replaced by MobileViT, which was a hybrid lightweight architecture that combined convolutional neural networks and Transformer components. The model complexity was significantly reduced to capture the local texture features and global semantic information, in order to recognize the small-scale and irregular lodging areas in cluttered environments. Secondly, the atrous spatial pyramid pooling (ASPP) module was restructured using depthwise separable Atrous convolutions and an optimized dilation rate scheme. Furthermore, the multi-scale contextual features were fused to reduce the computational cost. Thirdly, a convolutional block attention module (CBAM) was embedded in the following shallow feature extraction on the subtle edge and boundary information using channel and spatial attention mechanisms. Experimental results demonstrated that the MVDC-DeepLabv3+ achieved excellent segmentation accuracy on the constructed dataset, with the mean intersection over union (mIoU) of 94.10%, a mean pixel accuracy (mPA) of 97.44%, an F1-score of 96.91%, a precision of 96.40%, and a recall of 97.45%. The performance was improved by 3.70, 2.79, 2.09, 3.05, and 1.00 percentage points, respectively, compared with the original DeepLabv3+. The total model size was reduced to 5.47 MB, which was 75.6% and 97.4% smaller than the MobileNetV2 and Xception backbones, respectively, suitable for deployment on the embedded and resource-constrained platforms. A comparative analysis was carried out with the mainstream semantic segmentation models, such as UNet, SegNet, BiSeNet, and PSPNet. The superior performance was achieved to accurately segment the small lodging areas and then delineate the complex boundaries. Additional tests were also conducted under low-light nighttime conditions and dusty environments. The model’s robustness was further validated after the test. The pixel errors of the lodging detection were below 1.50%, indicating the reliability under challenging visual conditions. Finally, the segmentation model was integrated into an online detection system with PyQt5 and a vehicle-mounted camera interface. The system was then achieved in an average inference speed of 14.23 frames per second, where the relative pixel error was maintained below 1%. The high segmentation accuracy, strong robustness, and real-time performance can offer an effective technical solution to the intelligent wheat harvesting in precision agriculture.