To investigate the significant effect of Y sites on the dielectric constant and dielectric loss of Y₂MgTiO₆ (YMT) ceramics and further optimize the dielectric properties, this study used a solid-state sintering method to dope Y sites with Nd and Sm ions in the Y₂MgTiO₆ solid solution range. X-ray diffraction (XRD) refinement and vibrational spectroscopy confirmed that the substitution of Y sites with Nd and Sm led to the formation of a Y₂MgTiO₆ solid solution. The sample exhibited a pure phase and dense microstructure, with an optimal Q×f value of 74,821 GHz for YMT Nd (x = 0.1) and 67,068 GHz for YMT Sm (x = 0.05). These enhanced quality factor (Q×f) values indicate an improvement in the microwave dielectric performance, which is crucial for high-quality signal transmission in communication devices. The main intrinsic factors affecting the material properties were determined by chemical bond parameters and lattice vibrations to elucidate the changes in the crystal structure and vibration modes. Y-doping has been proven to be an effective strategy for improving the crystal stability of the Y₂MgTiO₆ system. This study paves the way for the precise adjustment of microwave dielectric properties by strategically selecting doping sites in complex ceramics, ultimately contributing to the development of high-performance electronic components for advanced communication systems.