Abnormal behavior detection method based on multi-modal feature fusion

Aiming at the problems of false detection, missing detection and imbalance of positive and negative samples in video abnormal behavior detection, a multi-modal feature fusion method for abnormal behavior detection is proposed. Firstly, a cross-modal sensing module is designed, which uses the cross-attention mechanism for feature fusion to improve the expression ability of cross-modal data features, and reduces the number of network parameters by sharing parameter strategies. Then, the improved binary cross entropy loss function is used to train the network. In the training process, the weight is reduced dynamically for the easily distinguishable samples, and the larger weight is focused on the difficult to distinguish samples, which improves the processing ability of unbalanced and difficult to classify data and improves the detection accuracy of abnormal behavior. Finally, through the strategy of sample batch selection, more abnormal fragments are filtered out by statistical analysis method to effectively solve the problem of missing abnormal fragments selection. Tests were conducted on XD-Violence, Shanghai-Tech open data set and self-made data set. The AP value of XD-Violence data set reached 85.32%, and the AUC value of Shanghai-Tech data set and self-made data set were 96.84% and 81.73%, respectively. Experimental results fully prove the effectiveness and generalization ability of this method.