Unsupervised domain adaptation (UDA) aims to leverage labeled source domain knowledge to improve the target domain's performance. Source-free domain adaptation (SFDA), a recent research focus, addresses challenges such as data privacy by relying solely on a pretrained source model and unlabeled target data, thus eliminating the need for direct access to source domain data. Although many studies have proposed methods such as generating a source domain, using a proxy source domain, or using pseudo-label training, these approaches directly fine-tune the source model, overlooking the excessive bias towards the source domain data in SFDA. The source domain model contains numerous source domain-specific features, and directly updating it to shift towards the target domain is hindered by these domain-specific features. To address this issue, we propose an inverse distillation-based SFDA method. By constructing an initial target domain model, the method extracts pure target domain features and distills them back into the source model, facilitating a smoother transition towards the target domain. Additionally, it identifies stable and active target samples from both structural and scoring perspectives, applying distinct matching strategies for pseudo-label selection. Extensive experiments and ablation studies on public datasets (Digits, Office-31, Office-Home and VisDA-2017) demonstrate the superior performance of our approach in SFDA tasks.