Trajectory prediction of unmanned swarm systems, serving as the foundation for behavioral and intentional cognition, has attracted extensive attention and made considerable progress in adversarial research. The influence of heterogeneous interaction relationships and external factors is crucial for trajectory prediction. Consequently, this paper proposes the Heterogeneous Graph with External Embedding (HGEE) network. We model the latent variables as multi-layer heterogeneous graphs based on prior knowledge of different interaction relationships and propose a method for calculating edge embeddings for heterogeneous graphs. Furthermore, we introduce a method that combines external environmental feature with historical observational trajectory data as the input for the decoder, enabling the model to learn the impacts of obstacles, targets, and desired formations on trajectories. We demonstrate that our approach surpasses state-of-the-art models in interaction inference and trajectory prediction through experiments on our proposed formation datasets based on consensus theory, across five evaluation metrics.