A novel deep neural network was proposed to predict the growth of micro voids in single-crystal metal based on U-Net and Transformer in this paper. The dataset was constructed through molecular dynamics (MD) simulation results of a single-crystal copper atom model with initial double ellipsoidal voids. A data preprocessing scheme based on background mesh was proposed to perform local statistics on the simulation results. The information obtained from simulation results, such as void morphology, dislocation distribution, and von Mises effective stress, was converted into local statistics on the background mesh. These statistics were then converted into pixel matrix format as the input of the deep neural network. Multiple data samples can be generated from the results of one single MD simulation, which significantly reduces the computational resources required for dataset generation. The samples encompass typical stages of the void growth, which enables the network to capture key features and to facilitate data augmentation conveniently. The deep neural network model consists of four parts: U-Net composed of down-sampling and up-sampling networks, a generation model, a Query network model, and a regression prediction network. The model input includes both physical information and positional information. The output is the predicted physical information for the next time step. The loss function is a superposition of loss functions for each predicted variable. Numerical examples demonstrate that the aforementioned deep-learning method can accurately predict the global porosity ratio, dislocation density, and von Mises stress during growth of micro voids in single-crystal metal. The time for the network prediction can reach two orders of magnitude lower than that of MD simulation.