Optimizing Tensor-Train Decomposition for efficient edge AI: Accelerated decoding via GEMM and reshape minimization

Tensor-Train Decomposition (TTD) has emerged as a powerful mathematical framework for compressing neural network models in edge-oriented deployments, significantly reducing communication overhead between cloud environments and resource-constrained edge devices. However, its widespread adoption is hindered by the substantial computational overhead of decoding compressed parameters on edge hardware. In this paper, we experimentally demonstrate and mathematically validate that TTD achieves superior compression efficiency and better accuracy retention compared to conventional pruning methods, particularly when fine-tuning is impractical. To overcome the critical decoding bottleneck, we propose a mathematically rigorous yet hardware-aware optimization framework specifically tailored for efficient TTD-based deployments. Our approach leverages existing General Matrix Multiplication (GEMM) accelerators, commonly available in modern edge processors, to substantially accelerate the computationally intensive Einsum operations inherent in TTD decoding. Furthermore, we analytically identify redundant reshape operations between the decoding and inference stages, introducing a novel merging strategy that significantly reduces memory-bound overhead. Evaluations on a Field-Programmable Gate Array (FPGA)-based edge inference processor show substantial improvements, including a 3 $\times$ speedup in reshape operations and a 69.3% decrease in decoding time. By seamlessly integrating rigorous mathematical formulation, analytical justification, and practical hardware optimization, this work paves the way for the efficient real-world deployment of TTD-compressed models on edge devices.