An improved version of Polak-Ribière-Polyak conjugate gradient method with its applications in neural networks training

Due to their simplicity, low memory requirements, strong convergence properties, and ability to solve problems of high dimensions, the conjugate gradient (CG) methods are widely used to solve linear and non-linear unconstrained optimization problems. The Polak-Ribière-Polyak (PRP) is considered as one of the most efficient CG methods in practical computation. However, theoretically, its convergence properties are poor. Therefore, many variants of PRP with good numerical results and good convergence properties have been developed, such as Gilbert and Nocedal method (PRP ${}^{+}$), Wei-Yau-Liu method (WYL), and Yousif et al. method (OPRP). In this paper, based on PRP ${}^{+}$ and OPRP methods, we proposed another modified version of PRP that inherits all the convergence properties of PRP ${}^{+}$ and OPRP and has improved numerical results. To show the efficiency and robustness of the new modified method in practice, it was compared with PRP ${}^{+}$, WYL, and OPRP when they are all applied under the strong Wolfe line search. At the same time, the new method was applied in deep learning to obtain ideal parameters of some neural network (NN) models during the training process.