While object tracking has made significantly strides in accuracy, there remains a notable gap in attention toward speed. Most trackers prioritize achieving real-time speed on powerful GPUs, yet practical applications demand higher tracking speeds, particularly on edge platforms. In response to this need, we introduce HCAT, an efficient tracking method designed for high speeds across diverse devices while maintaining superior tracking accuracy. At the core of HCAT lies a hierarchical cross-attention transformer that mitigates the serial nature of the transformer-based tracking model. Additionally, to further enhance the tracker's speed, we propose a feature sparsification module crafted to sparsify template features, consequently reducing the computational load of the model. HCAT demonstrates remarkable speed alongside competitive performance. For instance, it attains 55 fps on the NVIDIA Jetson AGX Xavier edge device, coupled with a 76.6% AUC score on the TrackingNet benchmark. Furthermore, we have developed an enhanced version, HCAT-M, integrating a multi-template framework and an update head. This variant establishes new state-of-the-art performance for high-speed tracking. Code and models are available at https://github.com/chenxin-dlut/HCAT.