Optimization Strategies for Matrix-Vector Multiplication Based on Ascend NPU

General Matrix-Vector Multiplication (GEMV) is a core operation in the decoding phase of large language models, and its performance directly determines the inference efficiency of the models. However, existing Neural Processing Units (NPUs) mostly implement GEMV operations by invoking General Matrix Multiplication (GEMM) interfaces, which suffer from poor adaptability and low hardware resource utilization in small-scale dimension scenarios. This paper conducts research on hardware adaptation and performance optimization of GEMV operators based on the Ascend 910 B NPU. By leveraging the hardware features of the Ascend 910 B AI Core compute kernels, this paper designs fundamental GEMV implementation schemes adapted to the corresponding compute units, respectively based on the matrix operation unit (AIC) and the vector operation unit (AIV). To meet the hardware data layout requirements of AIC, this paper equivalently transforms the GEMV operation into matrix-transposevector multiplication to improve result write-back efficiency. To address the memory access characteristics of the AIV, differentiated computation flows are designed according to different data layouts to ensure the continuity of memory access and compute instructions. On this basis, this paper further proposes general optimization strategies: adopting a rectangular tiling mechanism to maintain memory access continuity and reduce instruction dispatch overhead, and employing secondary task partitioning to alleviate computational load imbalance. Additionally, this paper designs an operator fusion interface that supports the fusion of GEMV with elementwise operators, with performance overhead kept within 2% when fusion computation is completed on the AIV unit. Experimental results show that under three data precision scenarios—single-precision floating-point, half-precision floating-point, and 8-bit integer, the performance of the GEMV implemented and optimized in this study reaches 3.38 times, 2.94 times, and 23.95 times (including dequantization) that of the default GEMM interface of the Ascend CANN platform, respectively. This provides strong support for efficient deployment and inference of large models on NPU platforms.