Binary honeycomb monolayer AgTe are experimentally evident to possess notably Rashba effect. Effective modulation of Rashba splitting facilitates the application of monolayer AgTe in spintronic devices. Here, we systematically investigated the effects of Se atom substitution doping on the atomic structure and the magnitude of Rashba splitting within monolayer AgSe_xTe_1−x on the Ag(111) substrate, utilizing scanning tunneling microscopy (STM) imaging and first-principles calculations. In the monolayer AgTe, Se atoms can easily substitute for the Te atoms, and they exhibit a more pronounced electronic state than Te atoms. Following the deposition of additional Se atoms and the subsequent annealing process, the proportion of Se atom substitution gradually increases. Once the value of x exceeds 0.33, domain boundaries appear within the AgSe_xTe_1−x monolayer. The density functional theory (DFT) calculation results indicate that as the atomic radius decreases from Te to Se, the reduced nuclear Coulomb potential alters the surface geometry. As a result, this gives rise to changes in the in-plane potential gradient and a corresponding decline in the Rashba parameter. Our finding provides a new approach for modulating the atomic structure and Rashba splitting of monolayer AgTe, which paves the way for the utilization of monolayer AgTe in the fields of nanoelectronics and spintronics.