Multi-agent reinforcement learning (MARL) has proven its effectiveness in cooperative multi-agent systems (MASs) but still faces issues on the curse of dimensionality and learning efficiency. The main difficulty is caused by the strong inter-agent coupling nature embedded in an MARL problem, which is yet to be fully exploited in existing algorithms. In this work, we recognize a learning graph characterizing the dependence between individual rewards and individual policies. Then we propose a graph-based reward aggregation (GRA) method, which utilizes the inherent coupling relationship among agents to eliminate redundant information. Specifically, GRA passes information among cooperating agents through graph attention networks to obtain aggregated rewards that contribute to the fitting of the value function, making each agent learn a decentralized executable cooperation policy. In addition, we propose a variant of GRA, named GRA-decen, which achieves decentralized training and decentralized execution (DTDE) when each agent only has access to information of partial agents in the learning process. We conduct experiments in different environments and demonstrate the practicality and scalability of our algorithms.