Journal Home > Volume 28 , Issue 4
Tsinghua Science and Technology 2023, 28(4): 673-685 https://doi.org/10.26599/TST.2022.9010038
Open Access | Issue | Published: 06 January 2023

Developing a Physiological Signal-Based, Mean Threshold and Decision-Level Fusion Algorithm (PMD) for Emotion Recognition

Show Author's Information Qiuju Zhang1,  Hongtao Zhang2,  Keming Zhou2,  Le Zhang1,3( )
College of Computer Science, Sichuan University, Chengdu 610065, China
August International Ltd., Hoddesdon, EN11 0EE, UK
Key Laboratory of Systems Health Science of Zhejiang Province, Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Hangzhou 310024, China
Keywords:
electroencephalogram (EEG), emotion recognition, machine learning, multimodal fusion, peripheral physiological signals
Cite this article:
Zhang Q, Zhang H, Zhou K, et al. Developing a Physiological Signal-Based, Mean Threshold and Decision-Level Fusion Algorithm (PMD) for Emotion Recognition. Tsinghua Science and Technology, 2023, 28(4): 673-685. https://doi.org/10.26599/TST.2022.9010038
Download citation
EndNote(RIS)
BibTeX

445

Views

87

Downloads

Citations

11

Crossref

5

WoS

9

Scopus

0

CSCD

Abstract Full text Electronic supplementary material About this article

With the development of computers, artificial intelligence, and cognitive science, engagement in deep communication between humans and computers has become increasingly important. Therefore, affective computing is a current hot research topic. Thus, this study constructs a Physiological signal-based, Mean-threshold, and Decision-level fusion algorithm (PMD) to identify human emotional states. First, we select key features from electroencephalogram and peripheral physiological signals, and use the mean-value method to obtain the classification threshold of each participant and distinguish individual differences. Then, we employ Gaussian Naive Bayes (GNB), Linear Regression (LR), Support Vector Machine (SVM), and other classification methods to perform emotion recognition. Finally, we improve the classification accuracy by developing an ensemble model. The experimental results reveal that physiological signals are more suitable for emotion recognition than classical facial and speech signals. Our proposed mean-threshold method can solve the problem of individual differences to a certain extent, and the ensemble learning model we developed significantly outperforms other classification models, such as GNB and LR.


menu
Abstract
Full text
Outline
Electronic supplementary material
About this article

Developing a Physiological Signal-Based, Mean Threshold and Decision-Level Fusion Algorithm (PMD) for Emotion Recognition

Show Author's information Qiuju Zhang1, Hongtao Zhang2, Keming Zhou2, Le Zhang1,3( )
College of Computer Science, Sichuan University, Chengdu 610065, China
August International Ltd., Hoddesdon, EN11 0EE, UK
Key Laboratory of Systems Health Science of Zhejiang Province, Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Hangzhou 310024, China

Abstract

With the development of computers, artificial intelligence, and cognitive science, engagement in deep communication between humans and computers has become increasingly important. Therefore, affective computing is a current hot research topic. Thus, this study constructs a Physiological signal-based, Mean-threshold, and Decision-level fusion algorithm (PMD) to identify human emotional states. First, we select key features from electroencephalogram and peripheral physiological signals, and use the mean-value method to obtain the classification threshold of each participant and distinguish individual differences. Then, we employ Gaussian Naive Bayes (GNB), Linear Regression (LR), Support Vector Machine (SVM), and other classification methods to perform emotion recognition. Finally, we improve the classification accuracy by developing an ensemble model. The experimental results reveal that physiological signals are more suitable for emotion recognition than classical facial and speech signals. Our proposed mean-threshold method can solve the problem of individual differences to a certain extent, and the ensemble learning model we developed significantly outperforms other classification models, such as GNB and LR.

Keywords: electroencephalogram (EEG), emotion recognition, machine learning, multimodal fusion, peripheral physiological signals

References(52)

[1]
F. Karray, M. Alemzadeh, J. Abou Saleh, and M. N. Arab, Human-computer interaction: Overview on state of the art, Int. J. Smart Sens. Intell. Syst., vol. 1, no. 1, pp. 137–159, 2008.
DOI Google Scholar
[2]
P. Ekman and W. V. Friesen, Facial Action Coding System. Palo Alto, CA, USA: Consulting Psychologists Press, 1978.
DOI
[3]
J. E. Cahn, The generation of affect in synthesized speech, J. Am. Voice I/O Soc., vol. 8, no. 1, pp. 1–19, 1990.
Google Scholar
[4]
R. E. Jack and P. G. Schyns, The human face as a dynamic tool for social communication, Curr. Biol., vol. 25, no. 14, pp. R621–R634, 2015.
DOI Google Scholar
[5]
L. A. Zebrowitz, M. Kikuchi, and J. M. Fellous, Facial resemblance to emotions: Group differences, impression effects, and race stereotypes, J. Pers. Soc. Psychol., vol. 98, no. 2, pp. 175–189, 2010.
DOI Google Scholar
[6]
J. Zhang, Z. Yin, P. Chen, and S. Nichele, Emotion recognition using multi-modal data and machine learning techniques: A tutorial and review, Inform. Fusion, vol. 59, pp. 103–126, 2020.
DOI Google Scholar
[7]
S. Hamann and T. Canli, Individual differences in emotion processing, Curr. Opin. Neurobiol., vol. 14, no. 2, pp. 233–238, 2004.
DOI Google Scholar
[8]
S. Koelstra, A. Yazdani, M. Soleymani, C. Mühl, J. S. Lee, A. Nijholt, T. Pun, T. Ebrahimi, and I. Patras, Single trial classification of EEG and peripheral physiological signals for recognition of emotions induced by music videos, in Proc. Int. Conf. BI 2010, Toronto, Canada, 2010, pp. 89–100.
DOI Google Scholar
[9]
S. D. Kreibig, Autonomic nervous system activity in emotion: A review, Biol. Psychol., vol. 84, no. 3, pp. 394–421, 2010.
DOI Google Scholar
[10]
D. N. De Gere, The face of masking: Examining central tendencies and between-person variability in display management and display rule, PhD dissertation, University of Washington, Seattle, WA, USA, 2008.
[11]
D. P. Subha, P. K. Joseph, R. A. U, and C. M. Lim, EEG signal analysis: A survey, J. Med. Syst., vol. 34, no. 2, pp. 195–212, 2010.
DOI Google Scholar
[12]
L. Zhang, J. Li, K. Yin, Z. Jiang, T. Li, R. Hu, Z. Yu, H. Feng, and Y. Chen, Computed tomography angiography-based analysis of high-risk intracerebral haemorrhage patients by employing a mathematical model, BMC Bioinf., vol. 20, no. 7, p. 193, 2019.
DOI Google Scholar
[13]
L. Zhang, C. Zheng, T. Li, L. Xing, H. Zeng, T. Li, H. Yang, J. Cao, B. Chen, and Z. Zhou, Building up a robust risk mathematical platform to predict colorectal cancer, Complexity, vol. 2017, p. 8917258, 2017.
DOI Google Scholar
[14]
R. Du and H. J. Lee, Power spectral performance analysis of EEG during emotional auditory experiment, in Proc. 2014 Int. Conf. Audio, Language and Image Processing, Shanghai, China, 2014, pp. 64–68.
DOI Google Scholar
[15]
R. W. Picard, E. Vyzas, and J. Healey, Toward machine emotional intelligence: Analysis of affective physiological state, IEEE Trans. Pattern Anal. Mach. Intell., vol. 23, no. 10, pp. 1175–1191, 2001.
DOI Google Scholar
[16]
G. Udovičić, J. Derek, M. Russo, and M. Sikora, Wearable emotion recognition system based on GSR and PPG signals, in Proc. 2nd Int. Workshop on Multimedia for Personal Health and Health Care, Mountain View, CA, USA, 2017, pp. 53–59.
DOI Google Scholar
[17]
F. Song, D. Mei, and H. Li, Feature selection based on linear discriminant analysis, in Proc. 2010 Int. Conf. on Intelligent System Design and Engineering Application, Changsha, China, 2010, pp. 746–749.
DOI Google Scholar
[18]
S. Koelstra, C. Muhl, M. Soleymani, J. S. Lee, A. Yazdani, T. Ebrahimi, T. Pun, A. Nijholt, and I. Patras, DEAP: A database for emotion analysis; using physiological signals, IEEE Trans. Affect. Comput., vol. 3, no. 1, pp. 18–31, 2012.
DOI Google Scholar
[19]
G. D. Liu, Y. C. Li, W. Zhang, and L. Zhang, A brief review of artificial intelligence applications and algorithms for psychiatric disorders, Engineering, vol. 6, no. 4, pp. 462–467, 2020.
DOI Google Scholar
[20]
L. Zhang, W. Bai, N. Yuan, and Z. Du, Comprehensively benchmarking applications for detecting copy number variation, PLoS Comput. Biol., vol. 15, no. 5, p. e1007069, 2019.
DOI Google Scholar
[21]
N. Zhuang, Y. Zeng, L. Tong, C. Zhang, H. Zhang, and B. Yan, Emotion recognition from EEG signals using multidimensional information in EMD domain, BioMed Res. Int., vol. 2017, p. 8317357, 2017.
DOI Google Scholar
[22]
K. Takahashi, Remarks on SVM-based emotion recognition from multi-modal bio-potential signals, in Proc. 13th IEEE Int. Workshop on Robot and Human Interactive Communication (IEEE Catalog No. 04TH8759), Kurashiki, Japan, 2004, pp. 95–100.
Google Scholar
[23]
A. Ross and A. K. Jain, Fusion techniques in multibiometric systems, in Face Biometrics for Personal Identification, R. I. Hammoud, B. R. Abidi, M. A. Abidi, eds. Berlin, Germany: Springer, 2007, pp. 185–212.
[24]
M. Cerna and A. F. Harvey, The fundamentals of FFT-based signal analysis and measurement, Application Note 041, National Instruments, https://scholar.google.com/scholar?hl=zh-CN&as_sdt=0%2C5&q=The+fundamentals+of+FFT-based+signal+analysis+and+measurement&btnG=, 2000.
[25]
P. J. Lang, M. K. Greenwald, M. M. Bradley, and A. O. Hamm, Looking at pictures: Affective, facial, visceral, and behavioral reactions, Psychophysiology, vol. 30, no. 3, pp. 261–273, 1993.
DOI Google Scholar
[26]
R. W. Levenson, P. Ekman, and W. V. Friesen, Voluntary facial action generates emotion-specific autonomic nervous system activity, Psychophysiology, vol. 27, no. 4, pp. 363–384, 1990.
DOI Google Scholar
[27]
R. McCraty, M. Atkinson, W. A. Tiller, G. Rein, and A. D. Watkins, The effects of emotions on short-term power spectrum analysis of heart rate variability, Am. J. Cardiol., vol. 76, no. 14, pp. 1089–1093, 1995.
DOI Google Scholar
[28]
P. Rainville, A. Bechara, N. Naqvi, and A. R. Damasio, Basic emotions are associated with distinct patterns of cardiorespiratory activity, Int. J. Psychophysiol., vol. 61, no. 1, pp. 5–18, 2006.
DOI Google Scholar
[29]
L. Zhang, L. Zhang, Y. Guo, M. Xiao, L. Feng, C. Yang, G. Wang, and L. Ouyang, MCDB: A comprehensive curated mitotic catastrophe database for retrieval, protein sequence alignment, and target prediction, Acta Pharm. Sin. B, vol. 11, no. 10, pp. 3092–3104, 2021.
DOI Google Scholar
[30]
J. Gao, P. Liu, G. D. Liu, and L. Zhang, Robust needle localization and enhancement algorithm for ultrasound by deep learning and beam steering methods, J. Comput. Sci. Technol., vol. 36, no. 2, pp. 334–346, 2021.
DOI Google Scholar
[31]
L. Zhang, J. Lv, M. Xiao, L. Yang, and L. Zhang, Exploring the underlying mechanism of action of a traditional Chinese medicine formula, Youdujing ointment, for cervical cancer treatment, Quant. Biol., vol. 9, no. 3, pp. 292–303, 2021.
Google Scholar
[32]
T. Li, Z. Cheng, and L. Zhang, Developing a novel parameter estimation method for agent-based model in immune system simulation under the framework of history matching: A case study on influenza a virus infection, Int. J. Mol. Sci., vol. 18, no. 12, p. 2592, 2017.
DOI Google Scholar
[33]
V. Doma and M. Pirouz, A comparative analysis of machine learning methods for emotion recognition using EEG and peripheral physiological signals, J. Big Data, vol. 7, no. 1, p. 18, 2020.
DOI Google Scholar
[34]
J. Zhang, P. Chen, S. Nichele, and A. Yazidi, Emotion recognition using time-frequency analysis of EEG signals and machine learning, in Proc. 2019 IEEE Symp. Series on Computational Intelligence (SSCI), Xiamen, China, 2019, pp. 404–409.
DOI Google Scholar
[35]
H. K. Palo, M. N. Mohanty, and M. Chandra, Use of different features for emotion recognition using MLP network, in Computational Vision and Robotics, I. K. Sethi, ed. New Delhi, India: Springer, 2015, pp. 7–15.
DOI
[36]
H. Gao, Z. Yin, Z. Cao, and L. Zhang, Developing an agent-based drug model to investigate the synergistic effects of drug combinations, Molecules, vol. 22, no. 12, p. 2209, 2017.
DOI Google Scholar
[37]
W. Wu, L. Song, Y. Yang, J. Wang, H. Liu, and L. Zhang, Exploring the dynamics and interplay of human papillomavirus and cervical tumorigenesis by integrating biological data into a mathematical model, BMC Bioinf., vol. 21, no. 7, p. 152, 2020.
DOI Google Scholar
[38]
Y. Xia, C. Yang, N. Hu, Z. Yang, X. He, T. Li, and L. Zhang, Exploring the key genes and signaling transduction pathways related to the survival time of glioblastoma multiforme patients by a novel survival analysis model, BMC Genomics, vol. 18, no. 1, p. 950, 2017.
DOI Google Scholar
[39]
M. Xiao, G. Liu, J. Xie, Z. Dai, Z. Wei, Z. Ren, J. Yu, and L. Zhang, 2019nCoVAS: Developing the web service for epidemic transmission prediction, genome analysis, and psychological stress assessment for 2019-nCoV, IEEE/ACM Trans. Comput. Biol. Bioinform., vol. 18, no. 4, pp. 1250–1261, 2021.
DOI Google Scholar
[40]
Y. You, X. Ru, W. Lei, T. Li, M. Xiao, H. Zheng, Y. Chen, and L. Zhang, Developing the novel bioinformatics algorithms to systematically investigate the connections among survival time, key genes and proteins for Glioblastoma multiforme, BMC Bioinf., vol. 21, no. 13, p. 383, 2020.
DOI Google Scholar
[41]
L. Zhang, C. Fu, J. Li, Z. Zhao, Y. Hou, W. Zhou, and A. Fu, Discovery of a ruthenium complex for the theranosis of glioma through targeting the mitochondrial DNA with bioinformatic methods, Int. J. Mol. Sci., vol. 20, no. 18, p. 4643, 2019.
DOI Google Scholar
[42]
L. Zhang, G. Liu, M. Kong, T. Li, D. Wu, X. Zhou, C. Yang, L. Xia, Z. Yang, and L. Chen, Revealing dynamic regulations and the related key proteins of myeloma-initiating cells by integrating experimental data into a systems biological model, Bioinformatics, vol. 37, no. 11, pp. 1554–1561, 2021.
DOI Google Scholar
[43]
X. Li, D. Song, P. Zhang, G. Yu, Y. Hou, and B. Hu, Emotion recognition from multi-channel EEG data through convolutional recurrent neural network, in Proc. IEEE Int. Conf. on Bioinformatics and Biomedicine (BIBM), Shenzhen, China, 2016, pp. 352–359.
DOI Google Scholar
[44]
L. Zhang, M. Xiao, J. Zhou, and J. Yu, Lineage-associated underrepresented permutations (LAUPs) of mammalian genomic sequences based on a Jellyfish-based LAUPs analysis application (JBLA), Bioinformatics, vol. 34, no. 21, pp. 3624–3630, 2018.
DOI Google Scholar
[45]
L. Zhang, J. Zhao, H. Bi, X. Yang, Z. Zhang, Y. Su, Z. Li, L. Zhang, B. J. Sanderson, J. Liu, et al., Bioinformatic analysis of chromatin organization and biased expression of duplicated genes between two poplars with a common whole-genome duplication, Hortic. Res., vol. 8, p. 62, 2021.
DOI Google Scholar
[46]
W. Liu, W. L. Zheng, and B. L. Lu, Emotion recognition using multimodal deep learning, in Proc. 23rd Int. Conf. on Neural Information Processing, Kyoto, Japan, 2016, pp. 521–529.
DOI Google Scholar
[47]
J. Kim and E. André, Emotion recognition based on physiological changes in music listening, IEEE Trans. Pattern Anal. Mach. Intell., vol. 30, no. 12, pp. 2067–2083, 2008.
DOI Google Scholar
[48]
K. H. Kim, S. W. Bang, and S. R. Kim, Emotion recognition system using short-term monitoring of physiological signals, Med. Biol. Eng. Comput., vol. 42, no. 3, pp. 419–427, 2004.
DOI Google Scholar
[49]
S. N. Daimi and G. Saha, Classification of emotions induced by music videos and correlation with participants’ rating, Expert Syst. Appl., vol. 41, no. 13, pp. 6057–6065, 2014.
DOI Google Scholar
[50]
H. J. Yoon and S. Y. Chung, EEG-based emotion estimation using Bayesian weighted-log-posterior function and perceptron convergence algorithm, Comput. Biol. Med., vol. 43, no. 12, pp. 2230–2237, 2013.
DOI Google Scholar
[51]
X. Li, P. Zhang, D. Song, G. Yu, Y. Hou, and B. Hu, EEG based emotion identification using unsupervised deep feature learning, presented at SIGIR2015 Workshop on Neuro-Physiological Methods in IR Research, Santiago, Chile, 2015.
Google Scholar
[52]
J. Atkinson and D. Campos, Improving BCI-based emotion recognition by combining EEG feature selection and kernel classifiers, Expert Syst. Appl., vol. 47, pp. 35–41, 2016.
DOI Google Scholar
File
673-685ESM.pdf (131.4 KB)
Publication history
Copyright
Acknowledgements
Rights and permissions

Publication history

Received: 06 July 2022
Revised: 22 August 2022
Accepted: 12 September 2022
Published: 06 January 2023
Issue date: August 2023

Copyright

© The author(s) 2023.

Acknowledgements

This work was supported by the National Science and Technology Major Project (No. 2018ZX10201002).

Rights and permissions

The articles published in this open access journal are distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/).

Return