Apple leaf disease segmentation is critical for yield and quality preservation in what is globally one of the most economically significant fruit crops. Despite recent advances in deep learning, real-world orchard environments present three primary challenges: (1) low contrast between lesions and background textures, which hinders accurate localization; (2) leaf overlap and occlusion, leading to incomplete feature representation and increased false negatives; and (3) the inherent limitations of unimodal RGB imagery in capturing subtle pathological features, which constrains generalization and accuracy. To address these issues, we proposed Language-Infused Visual Mamba (LViM), a dual-path U-Net architecture that integrates Mamba and Transformer modules for semantic-visual feature fusion. LViM achieves robust segmentation in complex environments through three core innovations: (1) A U-shaped Multimodal Transformer (MTT) branch integrated with AMBERT, which leverages inter-modal semantic relationships to enhance textual feature extraction and provide high-level semantic cues, thereby improving lesion-background discriminability; (2) a U-shaped Visual State Space (VMamba) branch that employs 2D Selective Scanning (SS2D) and Visual State Space (VSS) blocks to capture global context and fine-grained details, mitigating the impact of occlusion; and (3) Cross-Attention Gate Fusion (CAGF) and Linguistic Cross-Nested (LCN) modules that facilitate efficient cross-modal alignment and hierarchical feature modeling to better identify subtle lesions. Experimental results demonstrate that LViM consistently outperforms the VM-UNet baseline, yielding improvements of 4.05% in Precision, 4.25% in Dice coefficient, 4.49% in mIoU, and 4.23% in Recall.