Transformers have revolutionized medical image segmentation. However, their complexity leads to high parameter counts, increased FLOating-Point operations (FLOPs), and greater memory consumption, making them unsuitable for deployment on medical devices with limited computational resources. To overcome these limitations, we propose Efficient Shared Attention Network (ESA-Net), a lightweight and efficient model that achieves a favorable balance between accuracy and efficiency. ESA-Net adopts an encoder–decoder architecture, where the encoder incorporates an ESA module. This module leverages Content-Aware Position Encoding (CAPE) to enhance contextual sensitivity during feature extraction. The lightweight multi-scale decoder, based entirely on All Multi-Layer Perceptrons (All-MLP), ensures efficient reconstruction of segmentation maps. Experiments on the Synapse, ISIC17, ISIC18, and ACDC datasets validate the effectiveness of ESA-Net in multimodal medical image segmentation. For instance, on the Synapse dataset, ESA-Net achieves a dice score of 80.10% and reduces the Hausdorff distance to 15.34 mm. Moreover, ESA-Net demonstrates superior parameter efficiency, utilizing only 46% of the parameters of the Swin_UMamba model while maintaining comparable accuracy. These results highlight ESA-Net as a practical and deployable solution for medical image segmentation in resource-constrained environments.