COIN: Collaborative interaction-aware multiagent reinforcement learning for self-driving systems

Multiagent self-driving (MASD) systems provide an effective solution for coordinating autonomous vehicles (AVs) to reduce congestion and enhance both safety and operational efficiency in future intelligent transportation systems (ITS). Multiagent reinforcement learning (MARL) has emerged as a promising approach for developing advanced end-to-end MASD systems. However, achieving efficient and safe collaboration in dynamic MASD systems remains a significant challenge in dense scenarios with complex agent interactions. To address this challenge, we propose a novel MARL framework, named collaborative interaction-aware (COIN) . Specifically, we develop a new counterfactual individual-global twin delayed deep deterministic (CIG-TD3) policy gradient algorithm, crafted in a “centralized training, decentralized execution (CTDE)” manner, which aims to jointly optimize the individual objectives (navigation) and the global objectives (collaboration) of agents. We further introduce a dual-level interaction-aware centralized critic architecture that captures both local pairwise interactions and global system-level dependencies, enabling more accurate global value estimation and improved credit assignment for collaborative policy learning. We conduct extensive simulation experiments in dense urban traffic environments, which demonstrate that COIN consistently outperforms other advanced baseline methods in both safety and efficiency across various system sizes. These results highlight its superiority in complex and dynamic MASD scenarios, as further validated through real-world robot demonstrations.