Selective emitters are crucial as the key component determining the energy conversion efficiency of radioisotope thermophotovoltaic (RTPV) systems. Developing selective emitter materials with high selective emissivity, high spectral efficiency and excellent high-temperature stability can effectively improve the energy conversion efficiency and service life of RTPV systems. To adjust the selective emissivity and spectral efficiency, a series of rare earth tantalate selective emitters (Er(Ta_1−xNb_x)O₄ (0 ≤ x ≤ 0.2)) matching GaSb batteries were prepared by high-temperature solid-state reaction and pressureless sintering method. The as-prepared Er(Ta_1−xNb_x)O₄ (0 ≤ x ≤ 0.2) ceramics exhibit high emissivity (49%–93%) in the selective band (1.40–1.60 μm), high spectral efficiency (59.46%–62.12%) and excellent high-temperature stability at 1400 °C. On one hand, doping Nb⁵⁺ into the B-site changes the crystal local structure symmetry around Er³⁺, which promotes the f–f transition of Er³⁺ and enhances the selective emission performance. On the other hand, doping Nb⁵⁺ ions into the B-site can alter the bandgap and oxygen vacancy concentration to suppress non-selective emissivity. Increasing the selective emissivity and reducing the non-selective emissivity is beneficial for improving the spectral efficiency of selective emitters. Hence, the selective emissivity and spectral efficiency of Er(Ta_1−xNb_x)O₄ (0 ≤ x ≤ 0.2) can be effectively enhanced through compositional design, providing a new strategy for developing selective emitter materials for RTPV applications.