Aiming at the problems of traditional manual grading of stropharia rugoso-annulata, such as high labor intensity, low efficiency, and poor consistency, an improved method based on MobileViT model was proposed. By designing multi-scale modules with adaptive branching, adding local and global feature fusion, and introducing dual attention modules, the feature extraction capability is improved and the model robustness is enhanced. The experimental results show that the average recognition accuracy of the improved XCA-MobileViT for the five levels of stropharia rugoso-annulata datasets on the experimental platform is 97.71%, which is 2.34% higher than that of the MobileViT model, and the number of parameters and computation decreased by 0.401 M and 140.2 M respectively. Through validation experiments on two publicly available datasets of mushrooms, it was found that the F1 score and accuracy of XCA-MobileViT exceeded other models compared and showed good generalization.