To solve collaborative tasks, Multi-Agent Systems (MAS) have been widely used because of their collaboration capabilities and flexibility. They have been applied in various fields, such as robotic collaboration and intelligent transportation systems. In multi-agent collaborative problems, coordinating the collaboration strategies between agents is the most challenging aspect. Traditional Multi-Agent Reinforcement Learning (MARL) methods attempt to transform the team objective into individual objectives for each agent through value decomposition. However, the value allocation mechanisms in existing methods struggle to accurately measure the contribution differences between agents. In this paper, we propose an MARL method based on structured coordination by leveraging the local interaction structure among agents. Depending on the application scenarios, we present explicit and implicit implementation approaches. The explicit method constructs an explicit collaboration graph based on the cooperation relationships between agents, and then uses Shapley values to assess the contribution of each agent in the collaboration graph. Additionally, we propose a method based on self-attention mechanisms to dynamically construct implicit interaction structures. Experimental results demonstrate that, compared to several state-of-the-art multi-agent collaboration algorithms, the two collaboration methods we proposed achieve significant performance advantages across multiple complex collaboration scenarios.