Perspective Issue
Unified Programming Models for Heterogeneous High-Performance Computers
Journal of Computer Science and Technology 2023, 38 (1): 211-218
Published: 28 February 2023
Abstract Collect

Unified programming models can effectively improve program portability on various heterogeneous high-performance computers. Existing unified programming models put a lot of effort to code portability but are still far from achieving good performance portability. In this paper, we present a preliminary design of a performance-portable unified programming model including four aspects: programming language, programming abstraction, compilation optimization, and scheduling system. Specifically, domain-specific languages introduce domain knowledge to decouple the optimizations for different applications and architectures. The unified programming abstraction unifies the common features of different architectures to support common optimizations. Multi-level compilation optimization enables comprehensive performance optimization based on multi-level intermediate representations. Resource-aware lightweight runtime scheduling system improves the resource utilization of heterogeneous computers. This is a perspective paper to show our viewpoints on programming models for emerging heterogeneous systems.

Special Section Issue
Efficient memory allocator for the New Generation Sunway supercomputer
Journal of Tsinghua University (Science and Technology) 2022, 62 (5): 943-951
Published: 15 May 2022
Abstract PDF (5.9 MB) Collect

Supercomputers provide enormous computing power for large applications. Traditional supercomputers have mainly targeted scientific computing problems. However, other applications have new requirements for the both supercomputer software and hardware designs. The New Generation Sunway supercomputer has an inefficient memory allocator when running in the dynamic mode. This study develops an efficient memory allocator, SWAlloc, that reduces the memory allocation time of the brain scale pretrained model training framework, BaGuaLu, by up to 75 839 times. Evaluations using PARSEC also show that SWAlloc can speed up the memory allocation by up to 51 times (36% on average). SWAlloc has been deployed on the New Generation Sunway supercomputer for use by various large applications, including SWPytorch and SWTensorFlow.

Open Access Issue
AIPerf: Automated Machine Learning as an AI-HPC Benchmark
Big Data Mining and Analytics 2021, 4 (3): 208-220
Published: 12 May 2021
Abstract PDF (10.3 MB) Collect

The plethora of complex Artificial Intelligence (AI) algorithms and available High-Performance Computing (HPC) power stimulates the expeditious development of AI components with heterogeneous designs. Consequently, the need for cross-stack performance benchmarking of AI-HPC systems has rapidly emerged. In particular, the de facto HPC benchmark, LINPACK, cannot reflect the AI computing power and input/output performance without a representative workload. Current popular AI benchmarks, such as MLPerf, have a fixed problem size and therefore limited scalability. To address these issues, we propose an end-to-end benchmark suite utilizing automated machine learning, which not only represents real AI scenarios, but also is auto-adaptively scalable to various scales of machines. We implement the algorithms in a highly parallel and flexible way to ensure the efficiency and optimization potential on diverse systems with customizable configurations. We utilize Operations Per Second (OPS), which is measured in an analytical and systematic approach, as a major metric to quantify the AI performance. We perform evaluations on various systems to ensure the benchmark’s stability and scalability, from 4 nodes with 32 NVIDIA Tesla T4 (56.1 Tera-OPS measured) up to 512 nodes with 4096 Huawei Ascend 910 (194.53 Peta-OPS measured), and the results show near-linear weak scalability. With a flexible workload and single metric, AIPerf can easily scale on and rank AI-HPC, providing a powerful benchmark suite for the coming supercomputing era.

Total 3