This paper proposes a cross-layer efficient phase-aware Transformer segmentation algorithm for colorectal polyps in order to address the issues of irregular shape of the lesion region, fuzzy edge contour, and high similarity with normal region, which result in the loss of detail information and mis-segmentation of the lesion region. Firstly, the pyramid vision Transformer encoder is used to extract the global semantic information and spatial details of the input feature map layer by layer, and to analyze the colorectal polyp lesion features at multiple scales; secondly, the polarized self-attention module is used to regressively predict the lesion features, and to deepen the correlation of the semantic information of the features;thirdly, the high-efficiency phase-aware module is designed to extract the global and local information to precisely. The final one is the cross-layer fusion and propagation module, which enhances the rate of advanced feature reuse by integrating the edge details. Experiments were conducted on five datasets: CVC-ClinicDB, Kvasir-SEG, ETIS-LaribPolypDB, CVC-ConlonDB, and CVC-T, achieving Dice coefficients of 0.940, 0.923, 0.801, 0.810, and 0.896, respectively. This demonstrates superior segmentation performance over existing networks such as CaraNet and MSRAFormer. Both colorectal polyp images with fuzzy edges and complicated spatial organization exhibit great segmentation accuracy, according to the evaluation results.