Journal Home > Volume 28 , Issue 4

With the development of computers, artificial intelligence, and cognitive science, engagement in deep communication between humans and computers has become increasingly important. Therefore, affective computing is a current hot research topic. Thus, this study constructs a Physiological signal-based, Mean-threshold, and Decision-level fusion algorithm (PMD) to identify human emotional states. First, we select key features from electroencephalogram and peripheral physiological signals, and use the mean-value method to obtain the classification threshold of each participant and distinguish individual differences. Then, we employ Gaussian Naive Bayes (GNB), Linear Regression (LR), Support Vector Machine (SVM), and other classification methods to perform emotion recognition. Finally, we improve the classification accuracy by developing an ensemble model. The experimental results reveal that physiological signals are more suitable for emotion recognition than classical facial and speech signals. Our proposed mean-threshold method can solve the problem of individual differences to a certain extent, and the ensemble learning model we developed significantly outperforms other classification models, such as GNB and LR.


menu
Abstract
Full text
Outline
Electronic supplementary material
About this article

Developing a Physiological Signal-Based, Mean Threshold and Decision-Level Fusion Algorithm (PMD) for Emotion Recognition

Show Author's information Qiuju Zhang1Hongtao Zhang2Keming Zhou2Le Zhang1,3( )
College of Computer Science, Sichuan University, Chengdu 610065, China
August International Ltd., Hoddesdon, EN11 0EE, UK
Key Laboratory of Systems Health Science of Zhejiang Province, Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Hangzhou 310024, China

Abstract

With the development of computers, artificial intelligence, and cognitive science, engagement in deep communication between humans and computers has become increasingly important. Therefore, affective computing is a current hot research topic. Thus, this study constructs a Physiological signal-based, Mean-threshold, and Decision-level fusion algorithm (PMD) to identify human emotional states. First, we select key features from electroencephalogram and peripheral physiological signals, and use the mean-value method to obtain the classification threshold of each participant and distinguish individual differences. Then, we employ Gaussian Naive Bayes (GNB), Linear Regression (LR), Support Vector Machine (SVM), and other classification methods to perform emotion recognition. Finally, we improve the classification accuracy by developing an ensemble model. The experimental results reveal that physiological signals are more suitable for emotion recognition than classical facial and speech signals. Our proposed mean-threshold method can solve the problem of individual differences to a certain extent, and the ensemble learning model we developed significantly outperforms other classification models, such as GNB and LR.

Keywords:

electroencephalogram (EEG), peripheral physiological signals, machine learning, emotion recognition, multimodal fusion
Received: 06 July 2022 Revised: 22 August 2022 Accepted: 12 September 2022 Published: 06 January 2023 Issue date: August 2023
References(52)
[1]
F. Karray, M. Alemzadeh, J. Abou Saleh, and M. N. Arab, Human-computer interaction: Overview on state of the art, Int. J. Smart Sens. Intell. Syst., vol. 1, no. 1, pp. 137–159, 2008.
[2]
P. Ekman and W. V. Friesen, Facial Action Coding System. Palo Alto, CA, USA: Consulting Psychologists Press, 1978.
DOI
[3]
J. E. Cahn, The generation of affect in synthesized speech, J. Am. Voice I/O Soc., vol. 8, no. 1, pp. 1–19, 1990.
[4]
R. E. Jack and P. G. Schyns, The human face as a dynamic tool for social communication, Curr. Biol., vol. 25, no. 14, pp. R621–R634, 2015.
[5]
L. A. Zebrowitz, M. Kikuchi, and J. M. Fellous, Facial resemblance to emotions: Group differences, impression effects, and race stereotypes, J. Pers. Soc. Psychol., vol. 98, no. 2, pp. 175–189, 2010.
[6]
J. Zhang, Z. Yin, P. Chen, and S. Nichele, Emotion recognition using multi-modal data and machine learning techniques: A tutorial and review, Inform. Fusion, vol. 59, pp. 103–126, 2020.
[7]
S. Hamann and T. Canli, Individual differences in emotion processing, Curr. Opin. Neurobiol., vol. 14, no. 2, pp. 233–238, 2004.
[8]
S. Koelstra, A. Yazdani, M. Soleymani, C. Mühl, J. S. Lee, A. Nijholt, T. Pun, T. Ebrahimi, and I. Patras, Single trial classification of EEG and peripheral physiological signals for recognition of emotions induced by music videos, in Proc. Int. Conf. BI 2010, Toronto, Canada, 2010, pp. 89–100.
[9]
S. D. Kreibig, Autonomic nervous system activity in emotion: A review, Biol. Psychol., vol. 84, no. 3, pp. 394–421, 2010.
[10]
D. N. De Gere, The face of masking: Examining central tendencies and between-person variability in display management and display rule, PhD dissertation, University of Washington, Seattle, WA, USA, 2008.
[11]
D. P. Subha, P. K. Joseph, R. A. U, and C. M. Lim, EEG signal analysis: A survey, J. Med. Syst., vol. 34, no. 2, pp. 195–212, 2010.
[12]
L. Zhang, J. Li, K. Yin, Z. Jiang, T. Li, R. Hu, Z. Yu, H. Feng, and Y. Chen, Computed tomography angiography-based analysis of high-risk intracerebral haemorrhage patients by employing a mathematical model, BMC Bioinf., vol. 20, no. 7, p. 193, 2019.
[13]
L. Zhang, C. Zheng, T. Li, L. Xing, H. Zeng, T. Li, H. Yang, J. Cao, B. Chen, and Z. Zhou, Building up a robust risk mathematical platform to predict colorectal cancer, Complexity, vol. 2017, p. 8917258, 2017.
[14]
R. Du and H. J. Lee, Power spectral performance analysis of EEG during emotional auditory experiment, in Proc. 2014 Int. Conf. Audio, Language and Image Processing, Shanghai, China, 2014, pp. 64–68.
[15]
R. W. Picard, E. Vyzas, and J. Healey, Toward machine emotional intelligence: Analysis of affective physiological state, IEEE Trans. Pattern Anal. Mach. Intell., vol. 23, no. 10, pp. 1175–1191, 2001.
[16]
G. Udovičić, J. Derek, M. Russo, and M. Sikora, Wearable emotion recognition system based on GSR and PPG signals, in Proc. 2nd Int. Workshop on Multimedia for Personal Health and Health Care, Mountain View, CA, USA, 2017, pp. 53–59.
[17]
F. Song, D. Mei, and H. Li, Feature selection based on linear discriminant analysis, in Proc. 2010 Int. Conf. on Intelligent System Design and Engineering Application, Changsha, China, 2010, pp. 746–749.
[18]
S. Koelstra, C. Muhl, M. Soleymani, J. S. Lee, A. Yazdani, T. Ebrahimi, T. Pun, A. Nijholt, and I. Patras, DEAP: A database for emotion analysis; using physiological signals, IEEE Trans. Affect. Comput., vol. 3, no. 1, pp. 18–31, 2012.
[19]
G. D. Liu, Y. C. Li, W. Zhang, and L. Zhang, A brief review of artificial intelligence applications and algorithms for psychiatric disorders, Engineering, vol. 6, no. 4, pp. 462–467, 2020.
[20]
L. Zhang, W. Bai, N. Yuan, and Z. Du, Comprehensively benchmarking applications for detecting copy number variation, PLoS Comput. Biol., vol. 15, no. 5, p. e1007069, 2019.
[21]
N. Zhuang, Y. Zeng, L. Tong, C. Zhang, H. Zhang, and B. Yan, Emotion recognition from EEG signals using multidimensional information in EMD domain, BioMed Res. Int., vol. 2017, p. 8317357, 2017.
[22]
K. Takahashi, Remarks on SVM-based emotion recognition from multi-modal bio-potential signals, in Proc. 13th IEEE Int. Workshop on Robot and Human Interactive Communication (IEEE Catalog No. 04TH8759), Kurashiki, Japan, 2004, pp. 95–100.
[23]
A. Ross and A. K. Jain, Fusion techniques in multibiometric systems, in Face Biometrics for Personal Identification, R. I. Hammoud, B. R. Abidi, M. A. Abidi, eds. Berlin, Germany: Springer, 2007, pp. 185–212.
[24]
M. Cerna and A. F. Harvey, The fundamentals of FFT-based signal analysis and measurement, Application Note 041, National Instruments, https://scholar.google.com/scholar?hl=zh-CN&as_sdt=0%2C5&q=The+fundamentals+of+FFT-based+signal+analysis+and+measurement&btnG=, 2000.
[25]
P. J. Lang, M. K. Greenwald, M. M. Bradley, and A. O. Hamm, Looking at pictures: Affective, facial, visceral, and behavioral reactions, Psychophysiology, vol. 30, no. 3, pp. 261–273, 1993.
[26]
R. W. Levenson, P. Ekman, and W. V. Friesen, Voluntary facial action generates emotion-specific autonomic nervous system activity, Psychophysiology, vol. 27, no. 4, pp. 363–384, 1990.
[27]
R. McCraty, M. Atkinson, W. A. Tiller, G. Rein, and A. D. Watkins, The effects of emotions on short-term power spectrum analysis of heart rate variability, Am. J. Cardiol., vol. 76, no. 14, pp. 1089–1093, 1995.
[28]
P. Rainville, A. Bechara, N. Naqvi, and A. R. Damasio, Basic emotions are associated with distinct patterns of cardiorespiratory activity, Int. J. Psychophysiol., vol. 61, no. 1, pp. 5–18, 2006.
[29]
L. Zhang, L. Zhang, Y. Guo, M. Xiao, L. Feng, C. Yang, G. Wang, and L. Ouyang, MCDB: A comprehensive curated mitotic catastrophe database for retrieval, protein sequence alignment, and target prediction, Acta Pharm. Sin. B, vol. 11, no. 10, pp. 3092–3104, 2021.
[30]
J. Gao, P. Liu, G. D. Liu, and L. Zhang, Robust needle localization and enhancement algorithm for ultrasound by deep learning and beam steering methods, J. Comput. Sci. Technol., vol. 36, no. 2, pp. 334–346, 2021.
[31]
L. Zhang, J. Lv, M. Xiao, L. Yang, and L. Zhang, Exploring the underlying mechanism of action of a traditional Chinese medicine formula, Youdujing ointment, for cervical cancer treatment, Quant. Biol., vol. 9, no. 3, pp. 292–303, 2021.
[32]
T. Li, Z. Cheng, and L. Zhang, Developing a novel parameter estimation method for agent-based model in immune system simulation under the framework of history matching: A case study on influenza a virus infection, Int. J. Mol. Sci., vol. 18, no. 12, p. 2592, 2017.
[33]
V. Doma and M. Pirouz, A comparative analysis of machine learning methods for emotion recognition using EEG and peripheral physiological signals, J. Big Data, vol. 7, no. 1, p. 18, 2020.
[34]
J. Zhang, P. Chen, S. Nichele, and A. Yazidi, Emotion recognition using time-frequency analysis of EEG signals and machine learning, in Proc. 2019 IEEE Symp. Series on Computational Intelligence (SSCI), Xiamen, China, 2019, pp. 404–409.
[35]
H. K. Palo, M. N. Mohanty, and M. Chandra, Use of different features for emotion recognition using MLP network, in Computational Vision and Robotics, I. K. Sethi, ed. New Delhi, India: Springer, 2015, pp. 7–15.
DOI
[36]
H. Gao, Z. Yin, Z. Cao, and L. Zhang, Developing an agent-based drug model to investigate the synergistic effects of drug combinations, Molecules, vol. 22, no. 12, p. 2209, 2017.
[37]
W. Wu, L. Song, Y. Yang, J. Wang, H. Liu, and L. Zhang, Exploring the dynamics and interplay of human papillomavirus and cervical tumorigenesis by integrating biological data into a mathematical model, BMC Bioinf., vol. 21, no. 7, p. 152, 2020.
[38]
Y. Xia, C. Yang, N. Hu, Z. Yang, X. He, T. Li, and L. Zhang, Exploring the key genes and signaling transduction pathways related to the survival time of glioblastoma multiforme patients by a novel survival analysis model, BMC Genomics, vol. 18, no. 1, p. 950, 2017.
[39]
M. Xiao, G. Liu, J. Xie, Z. Dai, Z. Wei, Z. Ren, J. Yu, and L. Zhang, 2019nCoVAS: Developing the web service for epidemic transmission prediction, genome analysis, and psychological stress assessment for 2019-nCoV, IEEE/ACM Trans. Comput. Biol. Bioinform., vol. 18, no. 4, pp. 1250–1261, 2021.
[40]
Y. You, X. Ru, W. Lei, T. Li, M. Xiao, H. Zheng, Y. Chen, and L. Zhang, Developing the novel bioinformatics algorithms to systematically investigate the connections among survival time, key genes and proteins for Glioblastoma multiforme, BMC Bioinf., vol. 21, no. 13, p. 383, 2020.
[41]
L. Zhang, C. Fu, J. Li, Z. Zhao, Y. Hou, W. Zhou, and A. Fu, Discovery of a ruthenium complex for the theranosis of glioma through targeting the mitochondrial DNA with bioinformatic methods, Int. J. Mol. Sci., vol. 20, no. 18, p. 4643, 2019.
[42]
L. Zhang, G. Liu, M. Kong, T. Li, D. Wu, X. Zhou, C. Yang, L. Xia, Z. Yang, and L. Chen, Revealing dynamic regulations and the related key proteins of myeloma-initiating cells by integrating experimental data into a systems biological model, Bioinformatics, vol. 37, no. 11, pp. 1554–1561, 2021.
[43]
X. Li, D. Song, P. Zhang, G. Yu, Y. Hou, and B. Hu, Emotion recognition from multi-channel EEG data through convolutional recurrent neural network, in Proc. IEEE Int. Conf. on Bioinformatics and Biomedicine (BIBM), Shenzhen, China, 2016, pp. 352–359.
[44]
L. Zhang, M. Xiao, J. Zhou, and J. Yu, Lineage-associated underrepresented permutations (LAUPs) of mammalian genomic sequences based on a Jellyfish-based LAUPs analysis application (JBLA), Bioinformatics, vol. 34, no. 21, pp. 3624–3630, 2018.
[45]
L. Zhang, J. Zhao, H. Bi, X. Yang, Z. Zhang, Y. Su, Z. Li, L. Zhang, B. J. Sanderson, J. Liu, et al., Bioinformatic analysis of chromatin organization and biased expression of duplicated genes between two poplars with a common whole-genome duplication, Hortic. Res., vol. 8, p. 62, 2021.
[46]
W. Liu, W. L. Zheng, and B. L. Lu, Emotion recognition using multimodal deep learning, in Proc. 23rd Int. Conf. on Neural Information Processing, Kyoto, Japan, 2016, pp. 521–529.
[47]
J. Kim and E. André, Emotion recognition based on physiological changes in music listening, IEEE Trans. Pattern Anal. Mach. Intell., vol. 30, no. 12, pp. 2067–2083, 2008.
[48]
K. H. Kim, S. W. Bang, and S. R. Kim, Emotion recognition system using short-term monitoring of physiological signals, Med. Biol. Eng. Comput., vol. 42, no. 3, pp. 419–427, 2004.
[49]
S. N. Daimi and G. Saha, Classification of emotions induced by music videos and correlation with participants’ rating, Expert Syst. Appl., vol. 41, no. 13, pp. 6057–6065, 2014.
[50]
H. J. Yoon and S. Y. Chung, EEG-based emotion estimation using Bayesian weighted-log-posterior function and perceptron convergence algorithm, Comput. Biol. Med., vol. 43, no. 12, pp. 2230–2237, 2013.
[51]
X. Li, P. Zhang, D. Song, G. Yu, Y. Hou, and B. Hu, EEG based emotion identification using unsupervised deep feature learning, presented at SIGIR2015 Workshop on Neuro-Physiological Methods in IR Research, Santiago, Chile, 2015.
[52]
J. Atkinson and D. Campos, Improving BCI-based emotion recognition by combining EEG feature selection and kernel classifiers, Expert Syst. Appl., vol. 47, pp. 35–41, 2016.
File
673-685ESM.pdf (131.4 KB)
Publication history
Copyright
Acknowledgements
Rights and permissions

Publication history

Received: 06 July 2022
Revised: 22 August 2022
Accepted: 12 September 2022
Published: 06 January 2023
Issue date: August 2023

Copyright

© The author(s) 2023.

Acknowledgements

This work was supported by the National Science and Technology Major Project (No. 2018ZX10201002).

Rights and permissions

The articles published in this open access journal are distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/).

Return