An overlay multicast routing method based on network situational awareness and hierarchical multi-agent reinforcement learning

Compared with IP multicast, Overlay Multicast (OM) trees constructed at the application layer offer superior compatibility and flexible deployment advantages in heterogeneous, cross-domain networks. However, OM implementations under traditional network architectures suffer from weak adaptability to highly dynamic traffic due to their lack of awareness of underlying physical resource states. Moreover, reinforcement learning-based approaches fail to decouple the multi-objective tightly coupled nature of OM, resulting in high computational complexity, slow policy convergence, and insufficient stability. To address these challenges, we proposed a MA-DHRL-OM routing method. First, leveraging the centralized topological view provided by Software-Defined Networking (SDN), the method collected link-state information and constructed a traffic-aware feature model to provide multi-dimensional decision support for OM path planning. Second, within a unified framework that integrates multi-agent reinforcement learning and hierarchical reinforcement learning, MA-DHRL-OM solves for the optimal OM tree as follows: The hierarchical learning architecture decomposes the construction of the OM tree into a two-stage subtask framework. By designing tailored decision logic and reward signal feedback mechanisms for upper- and lower-layer agents, it achieved hierarchical decoupling of the high-dimensional OM problem, effectively reducing the action space dimensionality and enhancing policy convergence stability. Moreover, the multi-agent collaboration mechanism enabled each agent to make independent decisions based on its local observations, thereby balancing multi-objective optimization while improving the algorithm's overall scalability and adaptability. Extensive simulation experiments demonstrated that, compared with existing methods, MA-DHRL-OM achieves superior performance in optimizing key metrics such as delay, bandwidth utilization, and packet loss rate while exhibiting more stable convergence behavior and greater flexibility in OM routing decisions.