Object tracking using only visible-light sensor is usually problematic when facing some challenging scenes related to a complex environment like darkness, strong light, rain, fog, etc. In this paper, we have taken images, both thermal and visible light spectrum, from RGBT234 dataset as inputs for our proposed tracker which works as follows: First, the single shot multibox detector (SSD) is proposed to produce automatically the initial position of the target for visible and infrared modes. Second, a switching fuzzy logic controller (FLC) has been proposed to ensure automatic selection and switching between tracking modes (visible-light or thermal infrared spectrum) when a new environment is detected using confidence measures on some relevant features of the frames for each mode. Finally, to improve the kernelized correlation filter as a base tracker and overcome the limitation of the multimodal visual tracking, especially during tracking mode switching when visual information is not available, robust optimal smooth variable structure filter (SVSF) is proposed. Experiments on the recently public benchmark RGB-T234 demonstrate the effectiveness of our proposed method when compared to other state-of-the-art trackers.