Multi-Agent Reinforcement Learning (MARL) is an efficient cooperative training approach for adaptive traffic signal control (ATSC). With multiple agents seen as cooperative traffic intersections, and only be able to observe limited information in the real environment, agent policy space exploration is challenging. In addition, they share the environment reward, which makes it difficult to accurately measure contribution of individual agents. To tackle these problems, we propose the Graph Decomposition Action Reference (GDAR) framework based on Centralized Training Decentralized Execution (CTDE). Specifically, the multi-agent system is modeled as a graph structure, in which agents are regarded as nodes and relations as edges. To solve the problem of limited observation, Graph Neural Network (GNN) is used to expand the receiving domain of the agents. Meanwhile, we extract node representation to evaluate the individual contribution of each agent. In addition, we design action reference networks to improve the diversity of individual action choosing. We model the traffic conditions near the Nanjing Yangtze River Bridge in the Simulation of Urban MObility (SUMO). Experimental results show that GDAR adapts to ATSC tasks and is superior to advanced baselines.