Phosphogypsum (PG) and recycled concrete aggregates (RCA) are commonly used in base course materials as substitutes for natural aggregates. However, the use of PG and RCA alone may lead to issues such as poor water stability and insufficient durability. Therefore, this study considered using sodium metasilicate nonahydrate (SMN), calcium stearate emulsion (CSE), and emulsified styrene-butadiene rubber (ESBR) as additives for preparing base course materials to improve their water stability and durability. Dry-wet cycling test, freeze-thaw cycling test, dry shrinkage experiment, and fatigue test were conducted to evaluate the effects of the adopted additives on the comprehensive performance of a calcined PG (CPG)/RCA base course material. The results indicate that with the addition of SMN and CSE, the 28-day strength of CPG/RCA-SC increases to 8.29 MPa. After five freeze–thaw cycles, the residual compressive strength ratio of CPG/RCA-SC reaches 72%, which is substantially higher than that observed for the PG/RCA (36.7%) and for the CPG/RCA (50.6%). Furthermore, the incorporation of SMN, CSE, and ESBR reduces the 180-day drying shrinkage strain from 1172.5 × 10⁻⁶ for the RCA to 570 × 10⁻⁶ for the CPG/RCA-SCE. In terms of fatigue performance, the k-value is markedly increased from 1.083 for PG/RCA to 418.287 for CPG/RCA-SCE. Overall, the adopted additives can help largely consume PG and RCA into the base course for more sustainable and durable pavement applications.