As one of the common diseases of asphalt pavement, raveling can have a great adverse effect on the durability and road performance of asphalt pavement. To solve the problem of obtaining the asphalt-aggregate interface fracture energy required for asphalt pavement raveling prediction and sensitivity analysis, this research proposed a pendulum impact test method, by which the fracture energy of asphalt-aggregate interface was obtained by conducing lateral shear impact loading on “aggregate-asphalt-aggregate” specimens, causing a negligent failure at asphalt-aggregate interface. The research also analyzed the effects of temperature, asphalt type, aggregate type, water, and mineral filler on the asphalt-aggregate interface fracture energy. The results show that the asphalt-aggregate interface fracture energy can be used as the indicator to evaluate the asphalt-aggregate interface performance, reflecting the asphalt-aggregate adhesion performance, asphalt internal cohesion performance, and the mechanical characteristics of the asphalt-aggregate interface under different temperatures and loading conditions. Temperature has a great influence on the fracture energy of the asphalt-aggregate interface. As the temperature raises, the asphalt-aggregate interface fracture energy shows an upward trend followed by a downward trend, and the adhesion of asphalt and aggregate increases gradually, while the asphalt cohesion decreases. The fracture energy of SBS modified asphalt-aggregate interface and high-viscosity asphalt-aggregate interface are significantly greater than that of 70# asphalt-aggregate interface, while the difference between the fracture energy of the two modified asphalt and aggregate is not significant except for the peak value. The effect of aggregate types on the fracture energy of the asphalt-aggregate interface has no significant rule. Water will reduce the fracture energy of the asphalt-aggregate interface at all temperatures. The addition of mineral filler to asphalt can decrease the interface fracture energy and the range of optimum F/A ratio based on asphalt mixture performance do not have a positive effect on the asphalt-aggregate interface performance probably. The fracture energy of the asphalt-aggregate interface determined by the pendulum impact test can not only serve as a test basis for evaluating the performance of the asphalt-aggregate interface, but also provide material parameters for the numerical simulation analysis of raveling, and provide a criterion for the prediction of raveling and sensitivity analysis, so as to lay the foundation for the precise design of anti-raveling asphalt pavement.