In autonomous driving, target tracking is essential to environmental perception. The study of target tracking algorithms can improve the accuracy of an autonomous driving vehicle’s perception, which is of great significance in ensuring the safety of autonomous driving and promoting the landing of technical applications. This study focuses on the fusion tracking algorithm based on visible and infrared images. The proposed approach utilizes a feature-level image fusion method, dividing the tracking process into two components: image fusion and target tracking. An unsupervised network, Visible and Infrared image Fusion Network (VIF-net), is employed for visible and infrared image fusion in the image fusion part. In the target tracking part, Siamese Region Proposal Network (SiamRPN), based on deep learning, tracks the target with fused images. The fusion tracking algorithm is trained and evaluated on the visible infrared image dataset RGBT234. Experimental results demonstrate that the algorithm outperforms training networks solely based on visible images, proving that the fusion of visible and infrared images in the target tracking algorithm can improve the accuracy of the target tracking even if it is like tracking-based visual images. This improvement is also attributed to the algorithm’s ability to extract infrared image features, augmenting the target tracking accuracy.