ROFNet: Sheep behavior before lambing recognition with multimodel fusion of RGB and optical flow

Animal husbandry has been characterized by the intensification and scale in recent years. Some issues of animal welfare have emerged gradually, such as abnormal animal behavior and disease outbreaks. The high mortality rate of newborn lambs is one of the primary influencing factors on the development of the sheep industry. Prolonged parturition resulting from dystocia in ewes can lead to hypoxia and subsequent asphyxia in lambs, which stands as the main cause of newborn lamb deaths. Timely manual intervention is required to mitigate such risks. Therefore, the intelligent farming technologies can be expected to promptly and accurately identify animal behaviors for high breeding efficiency. Attention to the status of ewes approaching parturition and timely artificial assistance can also increase the survival rate of ewes and neonatal lambs. The prenatal behaviors of ewes in labor include the repetitive actions of standing up and then lying down, along with pawing the ground using their forehooves. In particular, the pawing the ground with the front hooves can differ from the daily behaviors; It is also difficult to recognize using conventional identification networks, because it is performed very rapidly. This study aims to develop a sheep behavior before lambing recognition network model (ROFNet) using the multimodal fusion of RGB and optical flow features, particularly with the help of motion information expressed by optical streams. The behavioral videos of the lambing ewes were captured and manually screened to identify the video segments of standing up, lying down, walking, and pawing the ground. The filtered videos were processed by frame-cutting. The middle frame of each video segment was selected as the representative frame for the RGB stream, thus providing for the overall spatial features. At the same time, an optical flow model was employed to obtain the optical flow data of the video frames. The color channels of the optical flow data were altered and superimposed to provide the temporal features of motion. ROFNet adopted a dual-stream network architecture to extract the spatial and temporal features using the RGB representative frames and optical flow frames, respectively. The two features were then fused using the Attentional Feature Fusion (AFF) module. The experimental results show that the ROFNet outperformed the existing advanced methods in several key metrics, including precision (88.10%), recall (89.34%), F1 score (88.54%), and accuracy (92.07%). Visual analysis by t-distributed stochastic neighbor embedding(t-SNE) shows that the ROFNet performed best in the confusable behaviors. The CAM heatmap further demonstrated that the ROFNet was used to focus more strongly on the front-leg region. A series of experiments demonstrates that the single modality was effectively avoided in the action recognition. Among them, the RGB stream and optical flow were used to extract the features via a dual-stream network architecture. The spatial features of RGB images were combined with the temporal features of optical flow images. Therefore, the ROFNet was achieved in the non-contact behavior recognition of the lambing ewes in labor using surveillance videos. The excellent performance was achieved in the accuracy and computational efficiency, indicating the great potential for practical application. The finding can also provide the technical support for the ewe parturition in intelligent livestock farming.