As the mainstream methods in current natural language processing and artificial intelligence, various pre-trained language models perform differently in the downstream tasks, due to their different language modeling, feature representation, model structure, training tasks and pre-training corpus, et al. In order to better integrate the knowledge in different pre-trained language models and utilize their learning abilities on the downstream tasks, we proposed a multi-model ensemble method MME-STS for semantic textual similarity justification tasks. The model structure and the corresponding feature representations were presented, and three different ensemble strategies based on average values, full-connected layer training and Adaboost algorithm with respect to model ensemble were also proposed. The effectiveness of MME-STS was also confirmed on two canonical benchmark datasets. Experimental results show that MME-STS outperforms single pre-trained language model-based approaches on the two benchmark datasets of SemEval 2014 task 4 SICK and SemEval 2017 STS-B corpus in terms of Pearson correlation coefficient and Spearman coefficient metrics.