Modern parallel and distributed computing systems are becoming increasingly complex as applications in high-performance computing (HPC) and artificial intelligence (AI) demand ever-greater levels of computation and communication efficiency. To address these demands, recent architectures integrate heterogeneous computing devices, such as CPUs, GPUs, and DPUs (or SmartNICs), within a single compute node, forming what we refer to as multi-rail heterogeneity. This trend offers substantial potential for scalability and performance but also amplifies the challenges of data movement, synchronization, and coordination across heterogeneous components. We propose Heterogeneity-Enriched Communication (HEC) as a new paradigm that embraces multi-rail heterogeneity by accurately analyzing communication primitives, adaptively composing multi-rail strategies, and scalably optimizing end-to-end pipelines. Through three representative case studies, including HCCL (collective communication), TrimEC (multi-rail erasure coding), and DPU-KV (edge data services), we demonstrate that HEC improves efficiency, scalability, and resilience in parallel and distributed systems for AI workloads. We envision HEC as a foundation for the next generation of AI infrastructure, harmonizing heterogeneous computing instruments into a symphony of scalable and efficient systems tailored for the emerging AI era.

Machine learning techniques have become ubiquitous both in industry and academic applications. Increasing model sizes and training data volumes necessitate fast and efficient distributed training approaches. Collective communications greatly simplify inter- and intra-node data transfer and are an essential part of the distributed training process as information such as gradients must be shared between processing nodes. In this paper, we survey the current state-of-the-art collective communication libraries (namely xCCL, including NCCL, oneCCL, RCCL, MSCCL, ACCL, and Gloo), with a focus on the industry-led ones for deep learning workloads. We investigate the design features of these xCCLs, discuss their use cases in the industry deep learning workloads, compare their performance with industry-made benchmarks (i.e., NCCL Tests and PARAM), and discuss key take-aways and interesting observations. We believe our survey sheds light on potential research directions of future designs for xCCLs.