$\mathtt{V}\mathtt{a}\mathtt{s}\mathtt{t}\mathtt{P}\mathtt{i}\mathtt{p}\mathtt{e}$: A High-Throughput Inference System via Adaptive Space-Division Multiplexing for Diverse Accelerators

The escalating demand on batched deep learning inference requires concurrent deployment of multiple deep neural network (DNN) models on a shared accelerator, thereby enabling spatial multiplexing to enhance resource utilization. Spatial multiplexing for co-locating multiple model services on the same accelerator increases the complexity of scheduling within a cluster. The meticulous collaborative optimization of model co-location combinations and resource allocation in a cluster creates an extensive configuration space for scheduling. In this paper, we present $\mathtt{V}\mathtt{a}\mathtt{s}\mathtt{t}\mathtt{P}\mathtt{i}\mathtt{p}\mathtt{e}$, a high-throughput inference system that schedules batch-oriented and heterogeneous requests on spatial multiplexing-enabled computing clusters. $\mathtt{V}\mathtt{a}\mathtt{s}\mathtt{t}\mathtt{P}\mathtt{i}\mathtt{p}\mathtt{e}$ determines optimal scheduling configurations by jointly optimizing model co-location and resource allocation using reinforcement learning to solve this combinatorial optimization problem. The experimental results demonstrate that on a large-scale cluster comprising 250 machine nodes with 1000 neural processing units (NPUs), $\mathtt{V}\mathtt{a}\mathtt{s}\mathtt{t}\mathtt{P}\mathtt{i}\mathtt{p}\mathtt{e}$ achieves average performance improvements of 2.2x, 1.3x, and 1.2x compared with the baseline systems, respectively. Furthermore, $\mathtt{V}\mathtt{a}\mathtt{s}\mathtt{t}\mathtt{P}\mathtt{i}\mathtt{p}\mathtt{e}$ is optimized and evaluated on mainstream GPUs. The results demonstrate that $\mathtt{V}\mathtt{a}\mathtt{s}\mathtt{t}\mathtt{P}\mathtt{i}\mathtt{p}\mathtt{e}$ achieves average throughput improvements of 2.7x on the NVIDIA A100 GPU and 1.9x on the AMD MI100 GPU.