Synthetic-augmented multimodal deep learning fuses dual-angle RGB images and phenology to unlock genotype-informative canopy structural trait in wheat

The wheat canopy genome harbors abundant yet untapped genetic variation that could be harnessed to enhance yield potential. The green area index (GAI) is a structural metric that reflects the photosynthetically active canopy surface and is closely linked to final grain yield. Current image-based GAI retrieval methods often suffer from signal saturation and coarse structural depiction, constraining downstream genetic analyses. To address this limitation, we constructed a comprehensive image dataset spanning eight field experiments across China and France, encompassing approximately 600 genotypes under six distinct management regimes. Leveraging this diverse data, we developed a multimodal deep-learning framework augmented by simulated-to-realistic (sim2real) synthetic data transfer. This framework fuses nadir and oblique RGB images with accumulated thermal time to produce high-precision, time-series GAI estimates. Validated on independent testing datasets from both China and France, the multimodal approach demonstrated robust performance with an accuracy of R² = 0.88 and an RMSE of 0.49 m² m⁻², representing an improvement of about 22% over the traditional gap fraction method. In three site-year field experiments involving 565 genotypes, the GAI dynamics derived from the multimodal approach showed higher broad-sense heritability (0.20-0.48) than those from the gap fraction approach (0.02-0.13) and stronger genotypic correlations with yield (0.19-0.40 versus 0.09-0.31). Furthermore, genetic analysis confirmed the biological fidelity of the estimated traits, identifying loci that co-localize with known architectural regulators such as Rht-D1, TaTB1-4D, and TaBGC1-4D. Consistently, the multimodal-derived phenotypes were specifically enriched in cell-wall remodeling and hormonal signaling pathways (e.g., brassinosteroid) that directly regulate canopy expansion. Overall, the proposed method offers a powerful tool for unlocking genetic gain in canopy architecture and accelerating canopy-targeted wheat improvement.