Detecting abnormal behavior is crucial for maintaining public security, especially in densely populated critical areas. Traditional target detection algorithms often struggle to deliver satisfactory results under these conditions due to challenges like dense target distribution, significant scale variation, and complex backgrounds. YOLOv8 is one of the better perforing detection models effect among the object detection models. This study introduces a novel approach to improve detection accuracy by integrating advanced mechanisms into the YOLOv8 backbone network. Firstly, the coordinate attention (CA) mechanism is incorporated into the C2f module of the backbone network. This enhances the network’s focus on targets amidst complex backgrounds by emphasizing relevant features and suppressing noise. Secondly, the swin transformer model is integrated into the YOLOv8 backbone. The swin transformer facilitates greater information interaction across the feature map, effectively utilizing the background information and improving object detection accuracy under complex scenarios. The datasets used in the experiments are described, the evaluation indexes of P, R, AP and mAP are listed, and ablation experiments and comparative experiments are carried out. Experiments demonstrate the feasibility and effectiveness of these improvements. The enhanced network is compared with several mainstream networks, showing a significant improvement in average accuracy, reaching 95.1%. Compared to the basic network YOLOv8, the average precision has been improved by 2.4%, which proves the effectiveness of this method. In summary, the innovative integration of the CA mechanism and Swin Transformer model into the YOLOv8 backbone network addresses key challenges in detecting abnormal behavior in densely populated and complex environments. These enhancements lead to improved detection accuracy, making it a promising approach for public security applications.