The exponential growth of Large Language Models (LLMs) drives transformative advances in Artificial Intelligence (AI). However, massive data demands of LLMs exceed the capacity of single Data Centers (DCs), requiring distributed frameworks across geographically dispersed facilities. While long-haul collaborative training across DCs is critical, challenges like latency and congestion in inter-DC communication remain under-addressed. To bridge this gap, we present the large-scale real-world validation of cross-DC LLM training using a 1024-GPU (Graphics Processing Unit) cluster across two DCs 120 km apart. We introduce the Long-haul Cross-DC Network (LCN), a communication framework integrating three engineering innovations: (1) an optimized Halving and Doubling (HD) algorithm which controls the communication traffic within the DC; (2) a dynamic user-level traffic management scheme that precisely adjusts server-specific traffic; and (3) a Fast Flow Pipeline (FFP) mechanism designed to prioritize critical packet forwarding. Experiments show that LCN limits the long-haul transmission performance degradation to 3%, outperforming baseline approaches without specialized optimization. Through the evaluation under different topologies, we show that LCN maintains a high training throughput while enabling cost-effective resource utilization. As the first production deployment of cross-DC LLM training, LCN solves the critical bottleneck in large-scale AI infrastructure, advancing sustainable advancements in distributed training.
Publications
- Article type
- Year
- Co-author
Article type
Year
Open Access
Research Article
Online First
Tsinghua Science and Technology
Published: 14 July 2026
Downloads:230
Total 1
京公网安备11010802044758号