Survey on topology of high-performance interconnection networks

In recent years, HPC (high-performance computing) has continued to develop rapidly, entering the post-exascale era and steadily advancing toward zettascale computing. With the rapid growth in intelligent computing demand driven by large language model applications, the scale of artificial intelligence data centers has expanded dramatically. High-performance interconnection networks are a key determinant of the scalability of both supercomputing and intelligent computing systems. The interconnection network topology, which defines the connection relationships among nodes, routers, and links, serves as the core of scalability-oriented design. The scalability design of a topology must simultaneously consider cost-effectiveness, high bandwidth, and low network diameter. The Fat-tree topology is the de facto standard in current interconnection networks, providing non-blocking bandwidth under arbitrary traffic patterns and delivering excellent overall performance. However, cost has always been a major constraint in interconnection network design. For zettascale computing and interconnecting systems involving over one hundred thousand accelerators, a non-blocking Fat-tree structure would incur interconnection costs of several billion dollars, which has motivated ongoing exploration in both academia and industry for more cost-efficient network topologies. Topology design is not only constrained by cost, but also by a variety of factors such as router chip architecture, physical packaging density, routing algorithm characteristics, power consumption limits, and fault tolerance. For example, when domestic router chips have limited port counts, constructing a topology that achieves high bandwidth, good scalability, and cost control presents a significant challenge. When the number of virtual channels is limited, implementing adaptive routing algorithms that are easy to deploy, accurate, and provide abundant path diversity is equally difficult. Furthermore, achieving fault tolerance in the face of link or switch failures is also a critical issue. These overlapping engineering constraints significantly increase the complexity of topology design.

According to their primary application scenarios, topologies can be categorized into three main classes: topologies for distributed data centers, topologies for high-performance parallel computing, and topologies for intelligent computing. Network topologies for distributed data centers mainly include the Fat-tree and its variants, DCell, BCube, and Jellyfish networks, all of which feature high bandwidth, scalability, and strong fault tolerance. Topologies for high-performance parallel computing are primarily high-dimensional Mesh/Torus networks based on low-radix routers, which have smaller bisection bandwidth and lower cost compared with Fat-trees, but suffer from larger network diameter and communication latency. In contrast, high-radix routers, under the constraint of a fixed total router chip bandwidth, can be designed to include a greater number of ports with lower per-port bandwidth. As network scale increases, adopting high-radix routers can significantly reduce network diameter, thereby lowering message transmission latency. Multiple topologies for HPC based on high-radix routers have been proposed, including Flattened Butterfly, Dragonfly, Dragonfly+, the Mesh-Tree topology in China’s Tianhe exascale prototype, Slim Fly, Galaxyfly, Bundlefly, Polarfly, and Polarstar. These topologies are often hierarchical and intra-dimensionally fully connected or constructed based on combinatorial or algebraic graph theory. They exhibit higher Moore-bound efficiency, smaller bisection bandwidth, and lower cost compared with Fat-tree networks, but rely heavily on efficient adaptive routing algorithms to realize their full performance potential — a key factor that limits their large-scale deployment. Topologies for intelligent computing mainly include HammingMesh, Google TPU, Rail-Only, HPN7.0, Zcube, and Zettafly. These networks are characterized by custom structural designs derived from the communication patterns of target applications, achieving optimized performance-to-cost ratios. This paper further summarizes the issues faced by current adaptive routing algorithms in high-radix networks dominated by Dragonfly, including indirect routing inefficiencies, phantom congestion, and asymmetric traffic patterns. It also provides a comprehensive comparison of different topologies in terms of network diameter, scalability, bisection bandwidth, and the size of the non-blocking region. Based on publicly available pricing information, this study models each topology’s per-endpoint copper/optical cable count, port count, cost, and power consumption, ultimately recommending the use of Zcube, Fat-tree, and Zettafly for building small-, medium-, and large-scale interconnection systems, respectively. The paper summarizes the main challenges in topology design, including cost optimization, routing efficiency, scalability, and achieving low network diameter.

Finally, the paper outlines future development trends in topology design. These include designing cost-optimal network topologies tailored to specific application characteristics; coordinating topology design with facility-level power delivery constraints; and the unification of intra-node network protocols together with the continual increase in supernode scale, leading to co-designed intra-node and inter-node topologies. In summary, network topology design is an engineering art of balance and compromise. It requires finding optimal solutions among multiple interdependent constraints, such as construction cost, power capacity, router port resource limitations, virtual channel constraints, efficient adaptive routing, and fault tolerance. Neither the lowest-cost topology nor the highest-performance, high-cost topology necessarily represents the optimal choice. The ideal topology should be highly aligned with the characteristics of its running applications, combining cost efficiency with sound design principles while maintaining simplicity in understanding, packaging, and deployment, and ensuring high manageability, stability, and reliability at the operational level.