In this study, fresh tea leaves were subjected to three withering processes: natural withering, far-infrared radiation for 3 h, and far-infrared radiation for 6 h. The contents of major taste substances were determined according to the Chinese national standards, and visible-near infrared (Vis-NIR) spectroscopy and machine vision (MV) data of the withered samples were collected to build an improved one-dimensional convolutional neural network model integrated with a convolutional block attention module (CBAM-1DCNN). The results showed that the phenol/ammonia ratio after infrared radiation for 3 h followed by natural withering for 15 h decreased by 20.06% compared with fresh leaves, and this treatment group achieved the highest sensory score. The CBAM-1DCNN model based on the Vis-NIR-MV fused data exhibited stronger discrimination capacity than did the models based on the Vis-NIR and MV data with an accuracy of 99.11% for the training set and 96.00% for the prediction set. Far-infrared radiation significantly altered the contents of major taste substances, and Vis-NIR spectroscopy combined with MV enabled rapid discrimination of the withering degree of black tea.