DRCU-Net: A 3D Segmentation Model for Rectal Tumors Based on Deeply Supervised Residual Capsule Networks

Existing Two-Dimensional (2D) rectal tumor segmentation methods struggle to fully exploit the relationships between slices due to the absence of Three-Dimensional (3D) spatial information, while traditional 3D segmentation techniques often suffer from poor performance and low rotational robustness. To address these limitations, we propose a 3D Magnetic Resonance (MR) segmentation model based on a deep supervised residual capsule network (namely DRCU-Net). This model introduces a capsule module built on the 3D U-Net architecture, allowing it to capture the spatial hierarchical features of tumor tissue through a dynamic routing mechanism, thereby enhancing the model’s robustness to rotation. Additionally, we employ deep supervision mechanisms to improve model performance and modify the LayerNorm function to extend layer normalization to 3D, facilitating its adaptation to the processing requirements of 3D data. We evaluate our model on the dataset from Shanxi Cancer Hospital (China), it achieves a Dice score of 0.7580 and a Mean Intersection over Union (MIoU) score of 0.6258, which demonstrating its superiority.