Social behavior in mice is critical for understanding their natural interactions and underlying neural mechanisms. Traditional Markov models, however, face limitations in capturing the sequential dynamics of body language associated with social behaviors. To address these challenges, we developed the body language-bidirectional encoder representation from transformers (BL-BERT) framework, which surpasses the Markov model in extracting complex sequential behavioral patterns. BL-BERT effectively differentiates the body language of the mice within different social interaction paradigms and produces results consistent with manual annotations. Notably, BL-BERT achieves higher extraction accuracy than the Markov model by reducing the complexity of the recurrent state transitions in behavior sequences. These advantages enable BL-BERT to accurately quantify high-order sequential behavioral structures in mice, paving the way for more detailed insights into the brain’s mechanisms controlling complex behavior.