In view of the characteristics of the small number and many types of rail defect samples, unstable transfer learning effect in real scenes, and threshold segmentation susceptible to environmental factors, an improved Markov defect segmentation method with zero samples was proposed. Firstly, the collected data was processed using Gabor functions to highlight defect features and reduce data dimensionality, resulting in a reduced dimensionality feature map. To mitigate the effects of glare and shadows, the feature map was subjected to Kmeans clustering to reduce data distribution. The clustering results were then used as a pre-classification matrix. Based on the reduced dimensionality feature map and the pre-classification matrix, a two-layer graph model of the improved Markov random field (MRF) was constructed for inference. The model analyzed the local geometric structure of the defect area by using the eigenvalues of the classification matrix inferred by the model. Finally, the defect regions were labelled, and the defect segmentation was completed. In the experimental part, a self-sampled dataset was used to draw the final conclusion through comparative and ablation experiments. The experimental results show that the proposed method achieves pixel accuracy, mean pixel accuracy, weighted intersection over union, and mean intersection over union of 93.6%, 80.7%, 89.4%, and 68.2%, respectively, on the self-sampled dataset, surpassing the accuracy of other comparative detection algorithms.