To address the problems of missing detection and low detection accuracy of the existing helmet wearing detection algorithms for small and crowded targets detection, this paper proposes a helmet wearing detection method based on improved YOLOv5 and transfer learning. First, different from the default priori frame that is not suitable for the task, we use the K-means algorithm to cluster the suitable priori frame size for the detection task. Then, in the back of the feature extraction network, we introduce a spatial channel mixed attention module to strengthen the learning of relevant weights and suppress the weights of irrelevant backgrounds, respectively. Further, we improve the judgment metric of the non-maximum-suppression (NMS) algorithm in the post-processing stage of YOLOv5 to reduce the phenomenon of false deletion and missing of prediction boxes. After that, the proposed network is trained based on the strategy of transfer learning, which can overcome the scarcity of limited existing data sets and improve the generalization ability of the model. Finally, we build a cascade judgment framework for helmet wearing deployed in visual sensor networks. The experimental results show that our proposed method improves the average accuracy (IOU=0.5) to 93.6%, which is 5% higher than the original model in the helmet wearing data set. The proposed model also outperforms other state-of-the-art algorithms by obviously improving the accuracy of helmet wearing detection in the construction scenarios.