Federated learning has emerged as a promising paradigm for privacy-preserving collaborative learning. Recently, personalized federated learning has attracted growing interest due to its ability to handle statistical heterogeneity across clients, such as hospitals or mobile devices. However, existing approaches often align local models with global representations that are domain-variant, biased toward dominant clients, or unstable during training, thereby compromising fairness and generalization. To address these limitations, we propose FedDTR, a novel personalized federated learning framework that leverages domain-invariant text-derived label embeddings as stable global references. These embeddings serve as global label anchors for cross-client alignment and provide intra-domain global priors. Specifically, FedDTR pulls sample embeddings toward their corresponding class anchors while pushing them away from those of other classes, thereby enhancing intra-class compactness and inter-class separability. Furthermore, an intra-domain prior module exploits local client data to estimate domain-specific global priors, enabling better modeling of the underlying data distribution. We evaluate FedDTR on benchmark datasets spanning computer vision and natural language processing, and demonstrate its consistent superiority over state-of-the-art methods.