Semi-supervised speaker verification system based on pre-trained models

In the evolving landscape of speaker verification (SV) systems, pre-trained models (PTMs) have become a cornerstone, significantly enhancing system performance through the integration of a speaker classification network and subsequent fine-tuning. Despite the advancements, current research mainly focuses on fine-tuning with labeled datasets, which poses a challenge owing to the necessity for a large amount of annotated data in the target domain. Therefore, this paper proposes a semi-supervised SV system leveraging PTMs, designed to excel under conditions of limited annotated data.

The proposed semi-supervised SV framework based on PTMs consists of several main steps. Initially, the entire model is fine-tuned using a small amount of labeled data, approximately 100 h, to create a high-performance seed model, referred to as model J. This model serves to extract speaker embeddings from a large unlabeled audio dataset. Using these embeddings, a speaker embedding graph is constructed, which is processed by the Infomap clustering module to generate pseudo-labels for each audio sample based on their clustering category. Next, the original labeled data is combined with the newly pseudo-labeled data for a comprehensive retraining from scratch, resulting in model B and marking the completion of the first iteration. Finally, the parameters of model C are fixed, and model C is then re-established as the seed model. The above steps are iteratively repeated to refine and develop the final SV system, referred to as model F.

The experiments, conducted on the VoxCeleb dataset, demonstrated the efficacy of the PTMs-based system in low-resource scenarios, specifically with 100 h of labeled VoxCeleb2 data. Notably, the semi-supervised framework showcased a remarkable improvement in speaker recognition performance, achieving a relative equal error rate (EER) reduction of 71.2% compared to the baseline SV system. Additionally, the semi-supervised system displayed competitive performance against fully supervised systems across all three VoxCeleb1 test sets, with EERs of 1.25%, 1.29%, and 2.45% on the VoxCeleb1-O/E/H test sets, respectively. Following the second iteration, a significant enhancement in performance could be observed, which, after subsequent iterations, began to converge, ultimately achieving an EER of 1.02% on the VoxCeleb1-O test set. Compared to the baseline system, the EER decreased by 86.8%.

This paper proposes a semi-supervised SV system utilizing PTMs tailored for scenarios with limited resources. By incorporating unlabeled audio data, the system leverages PTMs, the Infomap clustering algorithm, and a pseudo-label correction technique. The experimental results underscore the efficiency of the proposed semi-supervised training framework. Remarkably, even when restricted to merely 100 h of labeled data, the system achieves performance levels comparable to those of traditional fully supervised baseline systems. Furthermore, through multiple rounds of iterative training, a notable improvement in the system performance can be observed.