A new Sigma-Pi-Sigma neural network based on $L_1$ and $L_2$ regularization and applications

As one type of the important higher-order neural networks developed in the last decade, the Sigma-Pi-Sigma neural network has more powerful nonlinear mapping capabilities compared with other popular neural networks. This paper is concerned with a new Sigma-Pi-Sigma neural network based on a $L_1$ and $L_2$ regularization batch gradient method, and the numerical experiments for classification and regression problems prove that the proposed algorithm is effective and has better properties comparing with other classical penalization methods. The proposed model combines the sparse solution tendency of $L_1$ norm and the high benefits in efficiency of the $L_2$ norm, which can regulate the complexity of a network and prevent overfitting. Also, the numerical oscillation, induced by the non-differentiability of $L_1$ plus $L_2$ regularization at the origin, can be eliminated by a smoothing technique to approximate the objective function.