This paper examines the attitude and orbit tracking control problem of rigid spacecraft when there are uncertainties in both the irregular terms of the asteroid’s gravitational field and the spacecraft’s mass characteristic parameters. Based on the spacecraft’s prescribed performance error motion model described by Lie groups under irregular gravitational fields, a composite adaptive attitude and orbit tracking controller is proposed. A parameter update rule with higher convergence performance is devised based on the dynamic regression extension method and the immersion and invariance (I&I) theory to estimate the mass characteristic parameters with the goal of minimizing their uncertainty. Utilizing the estimated values of the mass characteristic parameters, an extended state observer is devised to estimate the overall system disturbance, which stems from the irregularity of the gravitational field and external disturbances. Building upon this parameter update law and disturbance estimation compensation, a composite adaptive prescribed performance terminal sliding mode controller is formulated. By using Lyapunov theory, it is demonstrated that the suggested controller guarantees that the errors in mass characteristic parameter estimate, disturbance estimation, and attitude and orbit tracking stay within a certain bound. Simulation results demonstrate that the incorporation of the dynamic regression extension method enhances the convergence performance of mass characteristic parameter estimation, and further, the addition of disturbance estimation compensation improves the accuracy of attitude and orbit tracking control.