As the mainstream methods in current natural language processing and artificial intelligence, various pre-trained language models perform differently in the downstream tasks, due to their different language modeling, feature representation, model structure, training tasks and pre-training corpus, et al. In order to better integrate the knowledge in different pre-trained language models and utilize their learning abilities on the downstream tasks, we proposed a multi-model ensemble method MME-STS for semantic textual similarity justification tasks. The model structure and the corresponding feature representations were presented, and three different ensemble strategies based on average values, full-connected layer training and Adaboost algorithm with respect to model ensemble were also proposed. The effectiveness of MME-STS was also confirmed on two canonical benchmark datasets. Experimental results show that MME-STS outperforms single pre-trained language model-based approaches on the two benchmark datasets of SemEval 2014 task 4 SICK and SemEval 2017 STS-B corpus in terms of Pearson correlation coefficient and Spearman coefficient metrics.

Although the pre-trained language models like BERT/RoBERTa/MacBERT can learn the grammatical, semantic and contextual features of characters and words well through the language mask model MLM pre-training task, they lack the ability to detect and correct spelling errors. What’s more, they faces the problem of inconsistency between the pre-training and downstream fine-tuning stages in Chinese spelling correction CSC task. In order to further improve BERT/RoBERTa/MacBERT’s ability of spelling error detection and correction, this paper proposed a self-supervised pre-training method MASC for CSC, which converts the prediction of masked words into recognition and correction of misspelled words on the basis of MLM. First of all, MASC expands the normal word-masking in MLM to whole word masking, aiming to improve BERT’s ability of learning semantic representation at word-level. Then, the masked words are replaced with candidate words from the aspects of the same tone, similar tone and similar shape with the help of external confusion set, and the training target is changed to recognize the correct words, thus enhancing BERT’s ability of detecting and correcting spelling errors. Finally, the experimental results on three open CSC corpora, sighan13, sighan14 and sighan15, show that MASC can further improve the effect of the pre-training language model, i. e. BERT/RoBERTA/MacBERT, in downstream CSC tasks without changing their structures. Ablation experiments also confirm the importance of whole word masking, phonetic and glyph information.