Random weights of DNNs and emergence of fixed points

We perform a numerical study of autoencoder deep neural networks (DNNs) when the input and the output vectors have the same dimension. Our focus is on fixed points (FPs) arising in these DNNs. We show that the existence and the number of these FPs depend on the distribution of randomly initialized DNNs' weight matrices. We first consider initialization with the identically and independently distributed (i.i.d.) light-tailed distributions of weights (e.g., Gaussian) and show existence of a single stable FP for a wide class of DNN architectures. In contrast, for heavy-tailed distributions (e.g., Cauchy), which typically appear after the training of DNNs, a number of stable FPs emerge. We observe an intriguing non-monotone dependence of the number of FPs on the DNN's depth. Finally, we link our result for untrained DNNs to the trained ones by showing that a number of FPs emerge after training of DNNs with light-tailed initialization.