A parallel resnet-transformer and cross-modal attention network for identification of rice blast resistance using UAV-based visible-infrared imagery

UAV-based visible-infrared image fusion can offer a promising tool for rapid identification of rice blast resistance (RBR), which is significant to promote rice breeding and ensure global food security. However, effectively combining the benefits of visible and infrared modalities for RBR identification still remains a challenge. This challenge stems from the difficulty of adaptively balancing the contribution of both modalities, and preserving the integrity of local and global representation for decision making. Also, the widely employed multi-scale feature fusion brings extra computational costs. To address the aforementioned limitations, this paper proposes a parallel resnet-transformer and cross-modal attention (PRCA) network for rapid RBR identification. The hybrid architecture processes the local and global feature learning in separate branches, retaining the representation ability of local and global features to maximum extent. Later, we introduce a visible-infrared cross-modal (VICM) attention mechanism to build the integration of visible and infrared modalities, striving to balance the contribution of visible and infrared imagery. Next, we present a lightweight multi-scale feature fusion (LMFF) method to further improve the precision, introducing no extra parameters and low computational costs. Comparison results of field tests on 530 rice plots showed that the PRCA model outperformed other SOTA methods in OA (+1.85%), kappa (+3.22%), F1-score (+0.24%), and recall (+1.38%). Finally, the effectiveness of our method was validated through ablation study. The experimental results indicate that the PRCA method has potential in RBR identification as well as quantitative inversion of other phenotypic traits.