The present work is concerned with the efficient numerical schemes for a time-fractional diffusion equation with tempered memory kernel. The numerical schemes are established by using a $L1$ difference scheme for generalized Caputo fractional derivative in the temporal variable, and applying the Legendre spectral collocation method for the spatial variable. The sum-of-exponential technique developed in [Jiang et al., Commun. Comput. Phys., 21 (2017), 650-678] is used to discrete generalized fractional derivative with exponential kernel. The stability and convergence of the semi-discrete and fully discrete schemes are strictly proved. Some numerical examples are shown to illustrate the theoretical results and the efficiency of the present methods for two-dimensional problems.