Accurate and automatic detection of pavement anomaly is critical for damage assessment and pavements maintainence. While existing Convolutional Neural Network (CNN) approaches have achieved high performance, their robustness to texture noise is limited, and the completeness of detected pixel-level cracks remains uncertain due to insufficient extraction of contextual information. To address these limitations, we propose a novel pavement anomaly detection network called RHCrackNet. This model incorporates feature fusion modules and feature enhancement modules to dynamically aggregate high-level semantic features with low-level detail features and enhance them through attention mechanisms. In addition, a non-local attention module is introduced to learn long-range dependencies and improve the connectivity of detected subtle cracks. To further enhance performance, auxiliary structure loss and direction loss are developed for supervised training. Experimental results show that RHCrackNet is highly competitive with state-of-the-art methods on six real-world datasets and has good generalization capabilities.