This paper studies the coupling mechanism between the nonlinear stiffness and damping coefficients of Active Elastic Support/Dry Friction Damper (AESDFD) and rotor system. First, parameters for evaluating the vibration reduction characteristics are proposed to facilitate the design of the AESDFD. To achieve this, the nonlinear friction force is initially represented as equivalent stiffness and damping coefficients, based on the ball-plate friction model. Second, three evaluation parameters—optimal slipping displacement, loss factor, and controllability—are proposed to reveal the vibration reduction characteristics of the AESDFD, alongside determining the optimal normal force. Subsequently, the finite element method, in conjunction with the ball-plate friction model, is introduced to formulate the governing equation of a low-pressure rotor system equipped with AESDFDs. The steady-state responses of the AESDFDs-rotor system are solved using the harmonic balance method combined with an efficient iteration method. Finally, the solutions are validated on the AESDFDs-rotor system both numerically and experimentally. The results indicate that controllability effectively assesses the vibration reduction performance of the AESDFD and is relatively insensitive to variations in low normal force. Away from the critical speed, the AESDFD suppresses vibration by altering the resonance position of the rotor system through its stiffness coefficient. Near the critical speed, vibration reduction is achieved primarily through energy dissipation by the damping coefficient. If the loss factor is less than one, the stiffness coefficient can diminish the vibration reduction effect of the damping coefficient. Notably, the optimal normal force of the AESDFD achieves optimal vibration reduction effect. This study reveals that changes in rotor system unbalance do not affect the vibration reduction characteristics of the AESDFD, with the same upper limit to the vibration reduction effect of the AESDFD.