Technologies for memory optimization for large model training on domestic platforms

The rapid evolution of LLMs (large language models) has led to an exponential increase in parameter counts, creating a severe contradiction with the relatively slow growth of GPU memory capacity—a phenomenon often referred to as the “Memory Wall.” For domestic computing platforms in China, this challenge is particularly acute. The significance of this research lies in its systematic focus on bridging the gap between high-compute requirements and limited memory resources within the context of indigenous hardware architectures such as Ascend, Cambricon, and MT-3000. Unlike global research that primarily targets general-purpose NVIDIA platforms, this work addresses the unique structural bottlenecks of domestic chips, including restricted PCIe bandwidth, customized instruction sets, and less mature software ecosystems. By analyzing recomputation and computation offloading strategies through the lens of hardware-software co-design, the paper provides a theoretical and practical framework for achieving “technological self-reliance” in AI training. The research is vital for the domestic industry as it enables the training of trillion-parameter models on local hardware, ensuring that Chinese AI development remains competitive despite international hardware constraints. It transforms memory optimization from a simple “space-saving” exercise into a strategic balancing act between computational efficiency, hardware adaptation, and system-level throughput, which is essential for the large-scale industrial deployment of domestic AI solutions.

Current research has progressed from single-technique breakthroughs to complex, multi-technology fusion strategies. The paper highlights significant milestones in mathematical programming for memory management, such as XEngine’s use of MIQP (mixed integer quadratic programming) and OLLA’s focus on minimizing tensor residence to reduce fragmentation. A major shift is observed in the transition from “offloading-heavy” approaches to more nuanced hybrid strategies. For instance, the DELTA scheme proposed by the NUDT (National University of Defense Technology) in 2024 optimizes dynamic control flow scenarios, breaking the traditional reliance on simple data swapping. Researchers have moved toward creating middleware that abstracts the core attributes of instruction sets (like LoongISA, SVE2, and BANG) into standardized parameters, allowing recomputation operators to adapt to different chips without redundant redesigns. Furthermore, the integration of hardware-native features—such as the Da Vinci architecture’s matrix units in Huawei Ascend—with dynamic compilation techniques has significantly improved memory access efficiency for long-sequence tasks. These advancements represent a move toward “platform-customized” optimization, where algorithms are no longer generic but are deeply coupled with the underlying hardware’s memory hierarchy and interconnect bandwidth.

The paper concludes that while domestic platforms face three primary hurdles—architectural heterogeneity, fragmented software ecosystems, and complex instruction modeling—these also present unique opportunities for “backward-compatible” innovation through software-defined hardware optimization. The systematic validation on platforms like MT-3000 proves that a “technology-platform-efficiency” argument chain is viable, providing a roadmap for chip manufacturers and framework developers. The researchers emphasize that the future of memory optimization lies in full-stack integration, where the boundaries between hardware scheduling, compiler optimization, and algorithmic sparsity are blurred. Moving forward, three key prospects are identified. First, the development of hardware-software synergistic memory management will focus on decoupling memory allocation from model compilation to handle massive sequence lengths. Second, the rise of MoE (mixture-of-experts) models necessitates system-level innovations in dynamic sparsity and load balancing to mitigate bandwidth limitations. Third, the growth of open-source ecosystems, exemplified by communities like DeepSeek, will be the primary catalyst for breaking the “software moat” of international competitors. The ultimate goal is to build a full-stack toolchain—covering development, deployment, and monitoring—that lowers migration costs for developers. By successfully “breaking the memory wall,” domestic platforms can transition from being functional alternatives to becoming high-performance leaders in the era of trillion-parameter model training.