In recent years, pretrained models have achieved remarkable progress in tasks such as speech recognition and speaker identification, significantly enhancing performance across a wide range of downstream applications, particularly when trained on large-scale speech datasets. However, the performance of pretrained models depends heavily on the quality of pseudolabels, and obtaining large quantities of audio data with true annotations is far more costly than acquiring unlabeled data. Therefore, improving the quality of pseudolabels generated from large-scale unlabeled data is critical. To address this challenge, this paper proposes a progressive clustering strategy to optimize pseudolabels produced by unsupervised clustering algorithms.

The progressive clustering process involved the following steps. First, HuBERT was trained on an unlabeled dataset, and the trained model was used to extract speaker embeddings from the audio data. Next, unsupervised clustering methods were applied to these embeddings to generate initial pseudolabels for each audio sample based on the clustering results. For each pseudolabel category, the mean of all speaker embeddings within that category was calculated as its centroid. Cosine similarity was then computed between the centroids of different categories. When the cosine similarity met a predefined threshold, the corresponding pseudolabel categories were merged to obtain a final, high-quality pseudolabel set.

Four algorithms, namely K-means, agglomerative hierarchical clustering, Infomap, and Leiden, were used to cluster the unlabeled Librispeech dataset to generate pseudolabels. After applying the progressive clustering strategy, the F₁-score, normalized mutual information, and cluster accuracy metrics of the pseudolabel set all improved, while the Coverage metric decreased. These results indicated that the quality of the pseudolabels improved significantly, demonstrating the positive effect of pseudolabel merging on clustering optimization. A slight decrease in purity was observed, which was attributed to the merging process inevitably combining some incorrectly assigned categories, thereby increasing intraclass complexity. This suggested that, in pursuit of higher category consistency and label accuracy, and in improving overall pseudolabel quality, some loss in cluster purity was unavoidable. Based on this strategy, a pretrained model incorporating joint speaker information modeling—speaker-related HuBERT (SR-HuBERT)—was trained. Experimental results showed that the model achieved a speaker identification accuracy of 85.95% and a speaker diarization error rate of 4.39% on speech processing universal PERformance benchmark speaker-related tasks, outperforming publicly available pretrained models. To further validate the generalization capability of the progressive clustering strategy, additional experiments were conducted on the VoxCeleb2 dataset, where pseudolabel quality also improved significantly after applying the strategy. The speaker representations extracted by SR-HuBERT were evaluated on VoxCeleb1-O, where the equal error rate metric decreased from 12.82% to 12.73%. These results indicate that the proposed strategy effectively enabled the model to learn more discriminative speaker representations.

This study proposes a progressive clustering strategy for unsupervised clustering to address the issue of pseudolabel quality in unlabeled audio data. The experimental results demonstrate that this strategy significantly improved pseudolabel quality. More importantly, under the guidance of higher-quality pseudolabels, model performance is further enhanced.

In the evolving landscape of speaker verification (SV) systems, pre-trained models (PTMs) have become a cornerstone, significantly enhancing system performance through the integration of a speaker classification network and subsequent fine-tuning. Despite the advancements, current research mainly focuses on fine-tuning with labeled datasets, which poses a challenge owing to the necessity for a large amount of annotated data in the target domain. Therefore, this paper proposes a semi-supervised SV system leveraging PTMs, designed to excel under conditions of limited annotated data.

The proposed semi-supervised SV framework based on PTMs consists of several main steps. Initially, the entire model is fine-tuned using a small amount of labeled data, approximately 100 h, to create a high-performance seed model, referred to as model J. This model serves to extract speaker embeddings from a large unlabeled audio dataset. Using these embeddings, a speaker embedding graph is constructed, which is processed by the Infomap clustering module to generate pseudo-labels for each audio sample based on their clustering category. Next, the original labeled data is combined with the newly pseudo-labeled data for a comprehensive retraining from scratch, resulting in model B and marking the completion of the first iteration. Finally, the parameters of model C are fixed, and model C is then re-established as the seed model. The above steps are iteratively repeated to refine and develop the final SV system, referred to as model F.

The experiments, conducted on the VoxCeleb dataset, demonstrated the efficacy of the PTMs-based system in low-resource scenarios, specifically with 100 h of labeled VoxCeleb2 data. Notably, the semi-supervised framework showcased a remarkable improvement in speaker recognition performance, achieving a relative equal error rate (EER) reduction of 71.2% compared to the baseline SV system. Additionally, the semi-supervised system displayed competitive performance against fully supervised systems across all three VoxCeleb1 test sets, with EERs of 1.25%, 1.29%, and 2.45% on the VoxCeleb1-O/E/H test sets, respectively. Following the second iteration, a significant enhancement in performance could be observed, which, after subsequent iterations, began to converge, ultimately achieving an EER of 1.02% on the VoxCeleb1-O test set. Compared to the baseline system, the EER decreased by 86.8%.

This paper proposes a semi-supervised SV system utilizing PTMs tailored for scenarios with limited resources. By incorporating unlabeled audio data, the system leverages PTMs, the Infomap clustering algorithm, and a pseudo-label correction technique. The experimental results underscore the efficiency of the proposed semi-supervised training framework. Remarkably, even when restricted to merely 100 h of labeled data, the system achieves performance levels comparable to those of traditional fully supervised baseline systems. Furthermore, through multiple rounds of iterative training, a notable improvement in the system performance can be observed.