The purpose of this paper is to investigate the impact of the varying population sizes on the dynamic behavior of the age-structured epidemic model. A age-structured SIRS epidemic model with the varying population sizes is established and investigated to take into account time delay. The non-negativity of the solution, the existence and stability of the steady states, and the existence of the Hopf bifurcation are discussed. The numerical simulations show that the varying population sizes can cause the age-structured SIRS model to produce multiple stability switches.