Multi-scale keypoints detection and motion features extraction in dairy cows using ResNet101-ASPP network

Detecting keypoints in dairy cows aims to locate and track the motion trajectories of the body’s joints, which plays a crucial role in behavior analysis and lameness detection. However, real farming scenarios, characterized by occlusions and large variations in object scale may result in poor detection results. Therefore, we introduce the atrous spatial pyramid pooling (ASPP) module into the shallow layers network of ResNet101, designed to improve the multi-scale feature extraction capability of the model. The ASPP module enhances the robustness of recognition for different dimensional sizes and occluded keypoints using different dilatation rates in the parallel atrous convolutional layers to expand the model’s receptive field. Furthermore, seven types of motion features, including tracking up, gait symmetry, step height balance, motion speed variability, head swing amplitude, head-neck slope and back curvature are extracted simultaneously by monitoring and tracking the motion trajectory of distinct keypoints. Several of these features represent innovative extraction models and attributes, first proposed in this study. Multiple models are trained and tested on datasets containing 2,385 frames for ablation experiments. The experiments show that, in comparison with the ResNet50, MobileNet_v2_1.0, and EfficientNet-b0 backbone networks, the training error and test error of ResNet101 are reduced by 4.04–30.12 pixels and 3.81–28.14 pixels. Therefore, ResNet101 is used as the benchmark for subsequent model improvement by adding the ASPP module. The training error and test error of the ResNet101-ASPP network are reduced by 0.27 and 0.24 pixels, respectively, compared to the benchmark network. The prediction confidence improves by 1.65–2.50% at three different dairy cow object scales. In addition, the keypoints under different occlusion conditions improve considerably, especially for small-scale keypoints, demonstrating the capability of the ASPP module for multi-scale feature extraction. By analyzing the distribution of the seven features and health, mild lameness, and severe lameness in dairy cows, it is shown that all the different features play an important role in distinguishing between different levels of lameness.