Adversarial attacks craft adversarial examples (AEs) to fool convolution neural networks. The mainstream gradient-based attacks, based on first-order optimization methods, encounter bottlenecks to generate high transferable AEs attacking unknown models. Considering that the high-order method would be a better optimization algorithm, we attempt to build high-order adversarial attacks to improve the transferability of AEs. However, solving the optimization problem of adversarial attacks directly via higher-order derivatives is computationally difficult and may face the non-convergence problem. So, we leverage the Runge−Kutta (RK) method, which is an accurate yet efficient high-order numerical solver of ordinary differential equation (ODE), to approximate high-order adversarial attacks. We first induce the gradient descent process of gradient-based attack as an ODE, and then numerically solve the ODE via RK method to develop approximated high-order adversarial attacks. Concretely, through ignoring the higher-order infinitesimal item in the Taylor expansion of the loss, the proposed method utilizes a linear combination of the present gradient and looking-ahead gradients to replace the computationally expensive high-order derivatives, and yields a relatively fast equivalent high-order adversarial attack. The proposed high-order adversarial attack can be extensively integrated with transferability augmentation methods to generate high transferable AEs. Extensive experiments demonstrate that the RK-based attacks exhibit higher transferability than the state of the arts.