Accurate classification is often required for the maize drought phenotypes from the in-field visible-light imagery for the irrigation decision support and yield-loss mitigation. Yet, the task remains challenging under practical farmland conditions. The scale variation, background clutter, and illumination fluctuations can jointly degrade the feature stability and inter-class separability. In this study, an improved Swin Transformer framework was developed for the nine-class phenotype classification over multiple growth stages and drought severities. A maize drought image dataset was established from multiple sources. The original 4 199 images were acquired from three channels, including the fixed high-definition cameras at agricultural stations, in-field smartphone photography, and publicly accessible online repositories. The 3 615 high-quality samples were obtained to remove the 584 low-quality or redundant images (13.9% of the originals) after manual screening. The dataset was split into the training, validation, and test subsets in a 7:1:2 ratio (2 530/362/723). Data augmentation—random rotations, salt-and-pepper noise injection, and adaptive histogram equalization—was applied only to the training set to enhance robustness and acquisition variability, expanding the training set to 3 628 images. Each image was annotated into one of nine categories defined by the intersection of three drought severity levels (no, mild, and severe drought) and three growth stages (jointing, tasseling, and maturity). In the backbone, Swin Transformer, a hierarchical Vision Transformer variant with shifted-window self-attention, was adopted to balance the contextual modeling and computational efficiency for the high-resolution field images. Three modifications of the baseline were incorporated at the Transformer block level. 1) Residual post-normalization: the layer normalization was placed after the residual connections to stabilize feature scaling in deep layers; 2) Scaled cosine attention: the dot-product similarity was replaced with the cosine similarity, and a learnable scaling factor was selected to reduce the sensitivity to activation magnitude for the similarity measurement under varying illumination; and 3) log-spaced continuous position bias (Log-CPB), the continuous relative position biases were generated via logarithmic coordinate mapping to facilitate the transfer over the different window sizes and image resolutions. In addition, a multi-scale dilated fusion attention (MDFA) module was designed to strengthen the drought-relevant representation. Four parallel dilated convolution branches (dilation rates 1, 6, 12, and 18) were integrated with the channel and spatial attention to adaptively emphasize the informative channels and locations associated with leaf curling, chlorosis/yellowing, and drying symptoms. The resulting model was termed the Swin Transformer with multi-scale dilated fusion attention (SWT-MDFA). Ablation experiments quantified the contribution rates of each component on the test set: The baseline Swin Transformer was achieved 95.9% accuracy, 95.4% precision, and 96.2% recall; The residual post-normalization increased the accuracy to 96.1%; The scaled cosine attention was improved the accuracy to 96.2% and recall to 96.6%; Log-CPB mainly benefited the recall to 96.8% under complex backgrounds; and MDFA was produced the largest single-module gain, thus reaching 97.0% accuracy with 96.3% precision and 97.2% recall. The SWT-MDFA was achieved in the best overall performance after components integration, with the 97.4% accuracy, 97.4% precision, and 97.3% recall, which were improved by 1.5, 2.0, and 1.1 percentage points over the baseline, respectively. Representative convolutional backbones—SE-ResNet, DenseNet-121, Xception, and Res2Net-50—confirmed that the superior performance of the SWT-MDFA was obtained over the high accuracy of Res2Net-50 (95.0%), Xception (93.6%), SE-ResNet (92.4%), and DenseNet-121 (91.2%). In terms of computational cost, the SWT-MDFA required 88.9 M parameters and 48.9 G floating-point operations (FLOPs), with substantial accuracy gains with a manageable computational budget. Gradient-weighted class activation mapping showed that the drought leaf regions suppressed the irrelevant background responses, indicating the interpretable and reliable phenotype classification at the growth stages and severity levels.