The evolution of metropolitan cities and increasing travel demand impose stringent requirements on traffic assignment methods. Multi-agent reinforcement learning (MARL) approaches outperform traditional methods in modeling adaptive routing behavior without requiring explicit system dynamics, making them attractive for real-world deployment. However, existing MARL frameworks face scalability and reliability challenges when managing large-scale networks with substantial and variable demand. This study proposes MARL-OD-DA, a novel framework that redefines agents as origin–destination (OD) pair routers and employs a continuous simplex-constrained action space. This reformulation reduces the agent population from O(N) (number of travelers) to O(|D|) (number of OD pairs), achieving at least two orders of magnitude fewer agents in practice while preserving convexity and enabling efficient adaptation to demand variation, thus significantly improving scalability. In contrast to prior MARL studies constrained to small-sized networks (up to 70 nodes, 2100 travelers) and fixed demand, MARL-OD-DA is validated on medium-sized networks (up to 416 nodes, 1406 OD pairs, and 360,600 travelers) under varying demand scenarios, demonstrating substantial improvements in scalability and applicability. To further enhance reliability, the framework integrates a Dirichlet-based policy, action pruning, and a relative gap-based reward. Theoretical analysis demonstrates that the Dirichlet-based policy reduces gradient bias, stabilizes variance, and enables sparse routing decisions, in contrast to the commonly used softmax-based policy. Experiments on three benchmark networks show that MARL-OD-DA significantly improves assignment quality and convergence speed. On the SiouxFalls network, the trained agents converge within 10 iterations during deployment, reducing the relative gap by 94.99% compared to conventional baselines.