Cross-Modal Knowledge Distillation for Depression Recognition: An Explainability Method with EEG and Pupil Area Signals

Multimodal physiological signals provide a more reliable data source for depression detection. For instance, combining electroencephalography (EEG) and pupil area (PA) signals can enhance depression recognition. However, EEG acquisition is challenging, limiting the practical use of EEG-based multimodal approaches, while PA signals are more accessible. Additionally, while existing explainability methods for time series models can quantify the contribution of each feature, they often fail to provide a comprehensive understanding of how these contributions drive performance improvements, limiting insights into the underlying mechanisms. To address these limitations and enhance the generalizability of PA-based depression detection models, this paper proposes a cross-modal knowledge distillation method, using an EEG and PA-based multimodal teacher model and a PA-based unimodal student model. Through knowledge distillation, complex multimodal features are transferred to the PA-based model, enhancing its performance. We also introduce Entropy-GradCAM (E-GCAM), an explainability method combining information entropy and gradient-weighted class activation mapping (Grad-CAM), to clarify mechanisms behind the student model’s performance gains. Quantitative results show that knowledge-distilled time series models encode more useful information, consistent with observed student model improvements. Experimental results demonstrate that the proposed method achieves optimal performance on two datasets, effectively reducing reliance on multimodal data and increasing the practicality of depression recognition models.