Due to the rising demand for Quality of Service (QoS) in the emerging network, this paper considers the routing problem of efficiently planning the transmission path of flow demands to balance the load of the network. Existing classical routing algorithms based on static decision-making are unable to adjust to dynamical scenarios with fluctuating flow. Traditional reinforcement learning-based methods face challenges in large-scale scenarios due to the vast state and action spaces. Besides, the neural networks used for policy generation also suffer from issues like rudimentary feature extraction and limited learning capability. In this paper, we first introduce RequestNet, a novel attention-based model designed to better leverage the information of network topology and feature correlations. We also propose a multi-agent modeling approach for the network packet routing problem, named RequestNet-MA, which treats pairs of edge routers as agents, allowing us to allocate traffic from the perspective of flow demands, thus significantly reducing the dimensions of state and action spaces and enhancing inter-agent communication and cooperation. Extensive experiments and ablation studies demonstrate that the proposed RequestNet-MA can reduce maximum link utilization and scale well to a large-scale network.