Targeting the low-efficiency problem of manual disease identification and disease mapping in the protection of calligraphy and painting cultural relics, this paper explores the feasibility of deep neural network-based object detection technology to identify calligraphy and painting diseases. We design a series of YOLOv5 models with some architecture optimizations based on the special requirements of disease identification. The optimizations include FGSM algorithm, CmBN strategy, Dropblock normalization, and CIOU_Loss loss function. Using the materials of calligraphy and painting of cultural relics in the museum as inputs, we enhance the images by combining Mosaic data enhancement method. Two disease identification deep learning models are trained based on some improvements, including sliding-window detection, image clipping based on layer-by-layer image analysis and positioning, etc. By evaluating the models with bench-mark performance metrics, this paper chooses YOLOv5x6 for our task. The experimental results show that YOLOv5x6 outperforms the other models with the best precision and recall. This model takes one-thousandth time compared with manual work. The introduction of deep learning techniques in disease identification not only helps to improve the efficiency of disease identification of cultural relics, but also provides objective and stable standards in the processes of disease identification.