In this paper, a fourth-order quasi-compact approximation for the normalized Riemann-Liouville tempered fractional derivatives was proposed. Its effectiveness was proved by using the generating function method, and it was applied to the numerical solution of the two-sided space tempered fractional diffusion equation with the time Caputo tempered fractional derivative. For the time Caputo tempered fractional derivative, we transformed the Caputo tempered fractional derivative into the Riemann-Liouville tempered fractional derivative through the relationship between them, and then employed the tempered weighted and shifted Grünwald difference operator to approximate the Riemann-Liouville tempered fractional derivative in the time direction. Thus, an efficient numerical scheme with second-order accuracy in time and fourth-order accuracy in space was derived. The stability and convergence of the numerical scheme were rigorously and elaborately proved, and the effectiveness of the numerical scheme was verified by a series of simulations conducted on numerical examples.