A biased stochastic three-term conjugate gradient algorithm with bandwidth-based step size for machine learning

In the field of machine learning, the solution of large-scale data optimization problems faces numerous challenges. Traditional conjugate gradient (CG) algorithms, though possessing excellent convergence properties, incur high computational costs when dealing with large-scale problems, thereby restricting their scope of application. On the other hand, stochastic gradient descent (SGD) algorithm, while being computationally inexpensive, has its convergence rates limited by the variance of gradient estimates, making it difficult to achieve satisfactory optimization results. To address these limitations, this paper proposes a biased stochastic three-term conjugate gradient algorithm. The proposed algorithm integrates the stochastic recursive gradient algorithm (SARAH) and a bandwidth-based step size strategy. Without incurring additional computational costs, it automatically incorporates upper and lower bounds on the step size, effectively balancing the flexibility and stability of the step size. Through the theoretical analysis presented in this paper, we demonstrate that the algorithm converges to a global optimum and analyze the linear convergence rate of the non-convex ( $\lambda$-gradient dominated) objective functions. The numerical results of two machine learning models demonstrate the strong competitiveness from the biased stochastic three-term conjugate gradient algorithm with bandwidth-based step size (SCGBW) algorithm.