Stochastic three-term conjugate gradient: a tensor correction algorithmic framework for nonconvex optimization in machine learning

In large-scale, nonconvex optimization for machine learning, such as the training of spine large models, the balance between efficient high-order geometric features of nonconvex landscapes identification and computational stability remains challenging. This paper proposes a tensor stochastic three-term conjugate gradient (TSTCG) algorithm to address two issues under a stochastic CG framework, the insufficient curvature awareness of the stochastic gradient descent and the heavy reliance on line searches. By introducing an implicit correction mechanism based on third-order tensor information, the algorithm fuses function values and gradient differences to reconstruct evolution vectors, effectively enhancing search precision in ill-conditioned curvature regions. Simultaneously, leveraging a specially designed three-term denominator structure allows the algorithm to process sufficient descent property, line search reliance reduction, and efficient fixed step-size updates. Finally, the nonconvexity proofs and numerical experiments demonstrate that TSTCG outperforms other mainstream stochastic optimization algorithms in terms of convergence efficiency and stability.