This study introduced an innovative numerical approach to examine combustion instability in Solid Rocket Motors (SRMs). The paper commenced with the derivation of a transient model for the solid propellant’s condensed phase, followed by its numerical discretization. Subsequently, this model was integrated with gas phase computations of the chamber’s internal flow field, encompassing fluid dynamics and combustion processes. The precision of the numerical method was validated by experimental data, and its reliability was confirmed through a grid independence analysis. The study then investigated the motor’s stability under various operating conditions, revealing the impact of parameters such as the sensitivity coefficient of the burning rate to temperature and the nozzle throat diameter on the motor’s stability. The results confirmed the bistable nature of combustion instability in specific regions. For instance, when the sensitivity coefficients of burning rate to ambient temperature (k₁) ranged from 1.4 to 1.8, the SRM adopted in this study with a throat diameter of 0.12 m remained stable under small disturbances but triggered instability under large disturbances. Moreover, increasing the value of k₁ and reducing the throat diameter can exacerbate combustion instability, leading to more pronounced nonlinear characteristics. The numerical method developed in this paper could effectively capture the nonlinear features of the combustion instability occurring in the motor, providing guidance for SRMs design.