A fatigued driving detection method using multimodal data fusion analysis

As a primary cause of road traffic injuries, fatigued driving requires efficient and accurate detection to improve traffic safety. Traditional single-signal approaches face limitations in capturing fatigue states, including high data collection intrusiveness, complex data structures, difficulty in real-time prediction, and low accuracy. Integrating information from different modalities has emerged as a new direction for development.

This study developed a multimodal fatigue detection model for drivers by using electrocardiogram signals, vehicle trajectory data, and driver facial video data collected during long-term real-vehicle driving experiments. For ECG signal processing, an improved two-step adaptive filtering method was adopted for denoising, followed by time-domain and frequency-domain analyses to extract the driver's ECG feature set. For vehicle trajectory data, the quartile method was first used to remove outliers, backward filling was applied to fill in missing values, and two-dimensional discrete wavelet analysis was then employed for data denoising. For facial image data, the LabelMe annotation tool was used to construct a personalized training dataset by manually annotating each driver's face with at least 300 images per subject. The YOLOv8 deep learning model was then fine-tuned and trained on this dataset, and the optimal model weights were saved. Next, the optimized model was used to automatically annotate the remaining unlabeled images, and the Hopenet algorithm was applied to extract head pose angles from the cropped facial regions. The model incorporated modules for data processing, feature extraction, feature selection, and fatigue prediction, utilizing a self-attention mechanism to capture long-term dependencies and generate predictive outputs.

The model achieved a maximum accuracy of 97.89% in predicting fatigue state categories, with overall recall and F1 scores exceeding 80%, demonstrating strong predictive accuracy. The detection model utilizing data from all three modalities served as the control group, while six experimental groups were formed using a single modality or a combination of two modalities. The experiments revealed that the fatigued driving detection model employing data from all three modalities achieved optimal performance across various metrics.

This study demonstrates that the proposed model successfully integrates information from different modalities, exhibits high accuracy and adaptability, and enhances the assurance of driving safety. Specifically, this model outperforms all comparison models in terms of accuracy, precision, recall, and F1 score, achieving the best performance in fatigued driving detection tasks, and maintains stable detection performance even under complex data structures, diverse sources, large time spans, average-quality facial images, and individual driver differences. The model achieves a maximum accuracy of 97.89% in identifying fatigue states, indicating that it correctly learns and recognizes fatigue patterns. Furthermore, models using a single modality or a combination of two modalities yield lower evaluation metrics than the three-modality model. Moreover, two-modality models consistently outperform single-modality variants, confirming that ECG signals, trajectory data, and facial images contribute positively to accurate fatigued driving detection.