A new hybrid conjugate gradient algorithm for solving the unconstrained optimization problem was presented. The algorithm could be considered as a modification of the memoryless Broyden-Fletcher-Goldfarb-Shanno (BFGS) quasi-Newton method. Based on a normalized gradient difference, we introduced a new combining conjugate gradient direction close to the direction of the memoryless BFGS quasi-Newton direction. It was shown that the search direction satisfied the sufficient descent property independent of the line search. For general nonlinear functions, the global convergence of the algorithm was proved under standard assumptions. Numerical experiments indicated a potential performance of the new algorithm, especially for solving the large-scale problems. In addition, the proposed method was used in practical application problems for image restoration and machine learning.

In this paper, aiming at the nonlinear equations, a new two-step Levenberg–Marquardt method was proposed. We presented a new Levenberg–Marquardt parameter to obtain the trial step. A new modified Metropolis criterion was used to adjust the upper bound of the approximate step. The convergence of the method was analyzed under the H $\ddot{\mathrm{o}}$lderian local error bound condition and the H $\ddot{}o$lderian continuity of the Jacobian. Numerical experiments showed that the new algorithm is effective and competitive in the numbers of functions, Jacobian evaluations and iterations.

First-order stochastic optimization algorithms have been widely applied to large-scale machine learning tasks. We have proposed a new stochastic optimization algorithm, which is inspired by an accelerated stochastic variance reduced method originally developed for convex optimization. The proposed algorithm addresses nonconvex and nonsmooth problems via the proximal operator and mitigates overfitting through a mini-batch strategy that exploits richer gradient information. We established sublinear convergence under standard assumptions. Extensive experiments on synthetic classification tasks, real-world datasets, nonconvex matrix factorization problems, and time series prediction tasks showed that the proposed algorithm consistently achieves faster convergence and competitive test performance compared to state-of-the-art stochastic optimization algorithms, underscoring its potential in practical applications for large-scale, nonsmooth learning problems.

In this paper, we propose a modified Levenberg–Marquardt (LM) method with a nonmonotone technique for solving nonlinear equations. Under the $\mathrm{H}\ddot{\mathrm{o}}\mathrm{l}\mathrm{d}\mathrm{e}\mathrm{r}\mathrm{i}\mathrm{a}\mathrm{n}$ continuity and the $\mathrm{H}\ddot{\mathrm{o}}\mathrm{l}\mathrm{d}\mathrm{e}\mathrm{r}\mathrm{i}\mathrm{a}\mathrm{n}$ local error bounds conditions, which are weaker than the local error bounds and the Lipschitz continuity, the global convergence and local convergence of the algorithm are proved. Numerical experiments also show that the algorithm is effective.