Breast mass identification is of great significance for early screening of breast cancer, while the existing detection methods have high missed and misdiagnosis rate for small masses. We propose a small target breast mass detection network named Residual asymmetric dilated convolution-Cross layer attention-Mean standard deviation adaptive selection-You Only Look Once (RCM-YOLO), which improves the identifiability of small masses by increasing the resolution of feature maps, adopts residual asymmetric dilated convolution to expand the receptive field and optimize the amount of parameters, and proposes the cross-layer attention that transfers the deep semantic information to the shallow layer as auxiliary information to obtain key feature locations. In the training process, we propose an adaptive positive sample selection algorithm to automatically select positive samples, which considers the statistical features of the intersection over union sets to ensure the validity of the training set and the detection accuracy of the model. To verify the performance of our model, we used public datasets to carry out the experiments. The results showed that the mean Average Precision (mAP) of RCM-YOLO reached 90.34%, compared with YOLOv5, the missed detection rate for small masses of RCM-YOLO was reduced to 11%, and the single detection time was reduced to 28 ms. The detection accuracy and speed can be effectively improved by strengthening the feature expression of small masses and the relationship between features. Our method can help doctors in batch screening of breast images, and significantly promote the detection rate of small masses and reduce misdiagnosis.