Comprehensive experimental and theoretical investigations on microstructure and mechanical properties were conducted to explore the refinement of Mg₂Si in Mg-Al-Si alloys through the addition of Ca and Y. Alloys of Mg-Al_5.5-Si_9.5-Ca_x (x = 0, 0.05, 0.1, 0.15, wt.%) and Mg-Al_5.5-Si_9.5-Y_x (x = 0, 0.3, 0.6, 0.9, wt.%) were designed based on the CALPHAD (CALculations of PHAse Diagram) calculations and literature data. With the experimentally determined optimal individual addition of 0.1 wt.% Ca and 0.9 wt.% Y respectively, a cross-experiment involving simultaneous addition of Ca and Y for the refinement of Mg₂Si was carried out. The alloy composition with the optimal refinement effect and mechanical properties was identified as Mg-Al_5.5-Si_9.5-Ca_0.1-Y_0.6 (wt.%). In Y-containing alloys, a novel core-shell structure evolution mechanism of MgSi₂Y₂//Al₄MgY//Mg₂Si was proposed based on experimental observation and first principles calculations, revealing MgSi₂Y₂ as the core for the heterogeneous nucleation of Mg₂Si.