Real-World Validation of Long-Haul Cross-Datacenter Network for Large Language Model Training

The exponential growth of Large Language Models (LLMs) drives transformative advances in Artificial Intelligence (AI). However, massive data demands of LLMs exceed the capacity of single Data Centers (DCs), requiring distributed frameworks across geographically dispersed facilities. While long-haul collaborative training across DCs is critical, challenges like latency and congestion in inter-DC communication remain under-addressed. To bridge this gap, we present the large-scale real-world validation of cross-DC LLM training using a 1024-GPU (Graphics Processing Unit) cluster across two DCs 120 km apart. We introduce the Long-haul Cross-DC Network (LCN), a communication framework integrating three engineering innovations: (1) an optimized Halving and Doubling (HD) algorithm which controls the communication traffic within the DC; (2) a dynamic user-level traffic management scheme that precisely adjusts server-specific traffic; and (3) a Fast Flow Pipeline (FFP) mechanism designed to prioritize critical packet forwarding. Experiments show that LCN limits the long-haul transmission performance degradation to 3%, outperforming baseline approaches without specialized optimization. Through the evaluation under different topologies, we show that LCN maintains a high training throughput while enabling cost-effective resource utilization. As the first production deployment of cross-DC LLM training, LCN solves the critical bottleneck in large-scale AI infrastructure, advancing sustainable advancements in distributed training.