To address the computational challenges posed by rapidly evolving intelligent applications, leading processor manufacturers are accelerating the integration of novel computing units into core processor architectures. Represented by Intel’s Advanced Matrix Extensions (AMX), the new generation of matrix computing units achieves a leap in computational performance compared with traditional vector units, such as Intel’s AVX-512, enabling single-chip peak performance to reach the order of hundreds of TFLOPS. These advanced computing units exhibit two notable characteristics: first, their efficient operation highly depends on specific data layouts; second, their demand for memory bandwidth is an order of magnitude higher than that of vector units. This architectural evolution renders traditional vector-unit-based optimization methods ineffective, highlighting the urgent need to explore new optimization strategies compatible with these units.

As a current research hotspot in the field of artificial intelligence, Mixture of Experts (MoE) has drawn significant attention from both academia and industry, making the acceleration of its inference a central focus. Matrix operations serve as the core computational components of MoE inference, with their performance directly determining overall inference efficiency. Specifically, the Grouped Query Attention (GQA) mechanism and the expert Feed-Forward Network (FFN) constitute the most critical performance bottlenecks in MoE model inference. Existing studies have conducted in-depth optimizations centered on these two components.

The choice to optimize matrix operations for CPUs equipped with AMX units is mainly based on the following three considerations. First, compared with GPUs, CPUs provide a more cost-effective solution for MoE inference. CPUs offer larger memory capacity, enabling the deployment of MoE models on a single node without relying on multiple GPUs. At the same time, CPU compute power and memory bandwidth are continuously improving, better meeting the performance requirements of MoE inference. Second, CPU memory bandwidth is significantly higher than the data transfer bandwidth between CPUs and GPUs. In inference scenarios with limited GPU resources, executing most matrix computations directly on the CPU often yields better performance. For example, the LIA framework demonstrates that when running large language model inference on a node equipped with only a single GPU, placing the primary computations on a CPU with AMX units significantly improves performance, especially when the input batch size is only a few hundred. Finally, compared with CPUs equipped only with vector units, AMX substantially enhances matrix operation performance. Moreover, although matrix optimization techniques for vector units are already well developed, optimization research for AMX still requires further exploration.

To fully exploit the performance of AMX, FlashMatrix was proposed, specifically designed to accelerate matrix operations in MoE. Its core consisted of two synergistic optimization strategies. First, it departed from the traditional oneDNN approach of performing layout transformations on weight matrices, instead applying layout transformations only to the significantly smaller input matrices in MoE, with these transformations executed within the cache and registers. This strategy greatly reduced the overhead of layout conversion while ensuring that weight matrices consistently maintained general layouts, such as row-major or column-major order. Second, a micro-kernel with the highest compute-to-memory-access ratio was designed to fully exploit AMX registers.

The experiments were conducted on Intel Gold 6430 and Intel Platinum 8468V, covering three core operators: single matrix multiplication, GQA, and expert FFN. The results show that FlashMatrix outperforms the state-of-the-art oneDNN library on all tested CPU platforms. In particular, for the performance of expert FFN, FlashMatrix achieves an average speedup of 2.5×. For end-to-end inference performance, FlashMatrix achieves a speedup of approximately 1.2×.

The two optimization strategies incorporated in FlashMatrix, namely the efficient layout transformation strategy and the micro-kernel with the highest compute-to-memory-access ratio, significantly enhance the performance of matrix operations in MoE. These optimizations provide efficient foundational operator support for deploying MoE models on CPUs.

In recent years, HPC (high-performance computing) has continued to develop rapidly, entering the post-exascale era and steadily advancing toward zettascale computing. With the rapid growth in intelligent computing demand driven by large language model applications, the scale of artificial intelligence data centers has expanded dramatically. High-performance interconnection networks are a key determinant of the scalability of both supercomputing and intelligent computing systems. The interconnection network topology, which defines the connection relationships among nodes, routers, and links, serves as the core of scalability-oriented design. The scalability design of a topology must simultaneously consider cost-effectiveness, high bandwidth, and low network diameter. The Fat-tree topology is the de facto standard in current interconnection networks, providing non-blocking bandwidth under arbitrary traffic patterns and delivering excellent overall performance. However, cost has always been a major constraint in interconnection network design. For zettascale computing and interconnecting systems involving over one hundred thousand accelerators, a non-blocking Fat-tree structure would incur interconnection costs of several billion dollars, which has motivated ongoing exploration in both academia and industry for more cost-efficient network topologies. Topology design is not only constrained by cost, but also by a variety of factors such as router chip architecture, physical packaging density, routing algorithm characteristics, power consumption limits, and fault tolerance. For example, when domestic router chips have limited port counts, constructing a topology that achieves high bandwidth, good scalability, and cost control presents a significant challenge. When the number of virtual channels is limited, implementing adaptive routing algorithms that are easy to deploy, accurate, and provide abundant path diversity is equally difficult. Furthermore, achieving fault tolerance in the face of link or switch failures is also a critical issue. These overlapping engineering constraints significantly increase the complexity of topology design.

According to their primary application scenarios, topologies can be categorized into three main classes: topologies for distributed data centers, topologies for high-performance parallel computing, and topologies for intelligent computing. Network topologies for distributed data centers mainly include the Fat-tree and its variants, DCell, BCube, and Jellyfish networks, all of which feature high bandwidth, scalability, and strong fault tolerance. Topologies for high-performance parallel computing are primarily high-dimensional Mesh/Torus networks based on low-radix routers, which have smaller bisection bandwidth and lower cost compared with Fat-trees, but suffer from larger network diameter and communication latency. In contrast, high-radix routers, under the constraint of a fixed total router chip bandwidth, can be designed to include a greater number of ports with lower per-port bandwidth. As network scale increases, adopting high-radix routers can significantly reduce network diameter, thereby lowering message transmission latency. Multiple topologies for HPC based on high-radix routers have been proposed, including Flattened Butterfly, Dragonfly, Dragonfly+, the Mesh-Tree topology in China’s Tianhe exascale prototype, Slim Fly, Galaxyfly, Bundlefly, Polarfly, and Polarstar. These topologies are often hierarchical and intra-dimensionally fully connected or constructed based on combinatorial or algebraic graph theory. They exhibit higher Moore-bound efficiency, smaller bisection bandwidth, and lower cost compared with Fat-tree networks, but rely heavily on efficient adaptive routing algorithms to realize their full performance potential — a key factor that limits their large-scale deployment. Topologies for intelligent computing mainly include HammingMesh, Google TPU, Rail-Only, HPN7.0, Zcube, and Zettafly. These networks are characterized by custom structural designs derived from the communication patterns of target applications, achieving optimized performance-to-cost ratios. This paper further summarizes the issues faced by current adaptive routing algorithms in high-radix networks dominated by Dragonfly, including indirect routing inefficiencies, phantom congestion, and asymmetric traffic patterns. It also provides a comprehensive comparison of different topologies in terms of network diameter, scalability, bisection bandwidth, and the size of the non-blocking region. Based on publicly available pricing information, this study models each topology’s per-endpoint copper/optical cable count, port count, cost, and power consumption, ultimately recommending the use of Zcube, Fat-tree, and Zettafly for building small-, medium-, and large-scale interconnection systems, respectively. The paper summarizes the main challenges in topology design, including cost optimization, routing efficiency, scalability, and achieving low network diameter.

Finally, the paper outlines future development trends in topology design. These include designing cost-optimal network topologies tailored to specific application characteristics; coordinating topology design with facility-level power delivery constraints; and the unification of intra-node network protocols together with the continual increase in supernode scale, leading to co-designed intra-node and inter-node topologies. In summary, network topology design is an engineering art of balance and compromise. It requires finding optimal solutions among multiple interdependent constraints, such as construction cost, power capacity, router port resource limitations, virtual channel constraints, efficient adaptive routing, and fault tolerance. Neither the lowest-cost topology nor the highest-performance, high-cost topology necessarily represents the optimal choice. The ideal topology should be highly aligned with the characteristics of its running applications, combining cost efficiency with sound design principles while maintaining simplicity in understanding, packaging, and deployment, and ensuring high manageability, stability, and reliability at the operational level.