Infrastructure construction in mountainous regions has led to notable challenges in managing waste soil disposal. Ultra-steep waste soil slopes, which exceed 100 m in height and have slope ratios steeper than 1:1, present substantial stability concerns under filling conditions. Conventional support systems often prove inadequate for these high-risk slopes. This study investigates the stability of an ultra-steep waste soil site in Xingshan County, Hubei Province, by proposing and evaluating a composite retaining system that combines anti-slide piles with reinforced soil. It aims to provide a scientifically validated solution to ensure safe construction and long-term stability of similar high-fill slopes, which are increasingly common but under-researched in geotechnical engineering.

A detailed three-dimensional finite element model is developed using Midas GTS NX to simulate complex geological conditions and the anti-slide pile-reinforced soil composite structure. The model accurately replicates the entire filling process across three main construction stages at elevations of 255, 315, and 350 m. The simulation incorporated appropriate constitutive models for various materials, interface elements to capture soil-structure interactions, and sequential activation/deactivation of elements to represent construction phases. Model parameters are calibrated with field geological survey data to ensure reliability and authenticity. The analysis comprehensively investigates the slope’s mechanical behavior by examining multiple parameters, namely displacement fields, plastic zone development, anti-slide pile deformation and internal forces (bending moment and shear force), and geogrid strain/axial force distribution along with their plastic states.

The result is revealed that the maximum horizontal displacement gradually shifted from the slope toe to the upper sections as filling progressed, with a peak value of 96.12 mm recorded at the interface between the third fill platform and the original slope. Notably, the plastic zones that is developed within the slope not merge into a continuous failure surface, demonstrating the system’s effectiveness in maintaining overall stability. Analysis of the anti-slide piles is showed that front-row piles consistently exhibits greater deformation than rear-row piles. The maximum bending moment of 5.57 MN·m is recorded in the outermost pile at a depth of 14 m at the rock-soil interface, indicating that the upper sections of the piles primarily resist landslide thrust, whereas the lower sections serve as anchorage. The geogrids is displayed a distinct strain distribution pattern, with maximum values found in the middle platform layers (approximately layers 7–12) and decreasing toward the upper and lower layers. The maximum tensile forces recorded are between 30% and 90% of the design reduced strength, confirming that all materials operated within their safe working limits.

The anti-slide pile-reinforced soil composite system effectively enhances the stability of ultra-steep waste soil slopes under filling conditions by controlling deformation and preventing progressive failure. This study illustrates the successful integration of rigid piles and flexible reinforcement, with critical load-bearing functions concentrated in the front-row piles and middle-platform geogrids. The findings offer valuable guidance for designing support systems in similar high-steep slope projects, emphasizing the need to strengthen the anti-bending capacity of these critical components. This study provides a foundation for safe and economical design approaches in challenging geotechnical engineering contexts.