A semantic segmentation method enhanced by multimodal collaboration in remote sensing foundation models

The integration of multi-modal remote sensing images greatly improves the comprehensiveness of land cover information. With the growing accessibility of multi-modal remote sensing data, multi-modal remote sensing foundation models are progressively developed to align different modalities, thereby facilitating the integration of diverse data sources. Nevertheless, existing foundation models typically concentrate on learning the common characteristics across modalities, forcing intrinsically irrelevant representations to converge while neglecting the synergistic information that is only revealed through modal interactions, thus hampering their advancement in comprehensive analysis of Earth observation data. To address this, we introduce UPSeg, a novel multi-modal remote sensing semantic segmentation method designed to explicitly capture and exploit these inter-modal synergies. UPSeg emulates human cognition by using unimodal features as inspiration for processing other modalities, effectively reducing uncertainties inherent in single-modality data. Specifically, we utilize features extracted from unimodal data as mutual inspiration and guidance to refine the feature parsing process of another modality, strengthening inter-modal interactions and maximizing synergistic benefits within the underlying model. Considering that the interaction of modality-unique information is more conducive to the generation of new insights, we propose a Variance Enhancement Module that employs a cross-modal attention mechanism to accentuate the distinctive features of each modality, enhancing the directionality and intentionality of cross-modal interactions. Extensive evaluation demonstrates that the proposed algorithm effectively addresses the limitations of existing foundation models in leveraging modal synergies, facilitating precise land cover classification. Our method outperforms the state-of-the-art multi-modal semantic segmentation algorithms with a gain of 2.0% in terms of Mean Pixel Accuracy (mPA) on WHU-OPT-SAR dataset and 4.2% on Gaofen Image Dataset (GID).