With the rapid construction of highway tunnels in China turns to the stage of both construction and maintenance, it will face the double pressure of the rapid growth of operating mileage and the deterioration of existing tunnels in the future, and the mobile detection and rapid diagnosis of structural safety have become research hotspots in the field of highway tunnel operation and maintenance. China has developed a lot of detection vehicles being used to tunnel apparent defects, which can provide an efficient method for rapid detection of tunnel surface cracks, water leakage, etc. However, highway tunnel lining images are characterized by complex backgrounds, multiple interference factors and low cracks occupancy, and accordingly, it’s difficult to do rapid identification and analysis of mass tunnel detection data, which has become the main bottleneck of technology promotion. Based on deep learning algorithm, this paper proposes a method that combines the target recognition with semantic segmentation algorithm. Firstly, Faster R-CNN deep neural network is used for target recognition on the original lining images to determine whether there are cracks and intelligently mark the crack regions within a rectangle. Then, the crack area selected by the frame is automatically cut, thereby filtering the pictures without cracks and removing the interference background in the pictures containing cracks, and then using the U-Net semantic segmentation network to segment the cracks at the pixel level. In practical engineering, it takes less than 0.15s to recognize the cracks in one image. Besides, this method can effectively identify various cracks in complex environments, with the F1 score of target recognition reaching 92% and the segmentation accuracy rate 98%. Compared with the previous feature recognition and global segmentation intelligent algorithms, this method significantly improves the speed and accuracy of recognition, and provides a feasible method for rapid and accurate recognition of mass defect detection data.