Journal Home > Volume 7 , Issue 3
Computational Visual Media 2021, 7(3): 407-417 https://doi.org/10.1007/s41095-021-0217-9
Research Article | Open Access | Issue | Published: 16 April 2021

Multi-scale joint feature network for micro-expression recognition

Show Author's Information Xinyu Li1,  Guangshun Wei1,  Jie Wang1,  Yuanfeng Zhou1( )
School of Software, Shandong University, Jinan 250101, China
Keywords:
deep learning, optical flow, micro-expression recognition, multi-scale feature
Cite this article:
Li X, Wei G, Wang J, et al. Multi-scale joint feature network for micro-expression recognition. Computational Visual Media, 2021, 7(3): 407-417. https://doi.org/10.1007/s41095-021-0217-9
Download citation
EndNote(RIS)
BibTeX

667

Views

22

Downloads

Citations

8

Crossref

5

WoS

8

Scopus

1

CSCD

Abstract Full text About this article

Micro-expression recognition is a substantivecross-study of psychology and computer science, and it has a wide range of applications (e.g., psychological and clinical diagnosis, emotional analysis, criminal investigation, etc.). However, the subtle and diverse changes in facial muscles make it difficult for existing methods to extract effective features, which limits the improvement of micro-expression recognition accuracy. Therefore, we propose a multi-scale joint feature networkbased on optical flow images for micro-expression recognition. First, we generate an optical flow image that reflects subtle facial motion information. The optical flow image is then fed into the multi-scale joint network for feature extraction and classification. The proposed joint feature module (JFM) integrates features from different layers, which is beneficial for the capture of micro-expression features with different amplitudes. To improve the recognition ability of the model, we also adopt a strategy for fusing the feature prediction results of the three JFMs with the backbone network. Our experimental results show that our method is superior to state-of-the-art methods on three benchmark datasets (SMIC, CASME II, and SAMM) and a combined dataset (3DB).


menu
Abstract
Full text
Outline
About this article

Multi-scale joint feature network for micro-expression recognition

Show Author's information Xinyu Li1, Guangshun Wei1, Jie Wang1, Yuanfeng Zhou1( )
School of Software, Shandong University, Jinan 250101, China

Abstract

Micro-expression recognition is a substantivecross-study of psychology and computer science, and it has a wide range of applications (e.g., psychological and clinical diagnosis, emotional analysis, criminal investigation, etc.). However, the subtle and diverse changes in facial muscles make it difficult for existing methods to extract effective features, which limits the improvement of micro-expression recognition accuracy. Therefore, we propose a multi-scale joint feature networkbased on optical flow images for micro-expression recognition. First, we generate an optical flow image that reflects subtle facial motion information. The optical flow image is then fed into the multi-scale joint network for feature extraction and classification. The proposed joint feature module (JFM) integrates features from different layers, which is beneficial for the capture of micro-expression features with different amplitudes. To improve the recognition ability of the model, we also adopt a strategy for fusing the feature prediction results of the three JFMs with the backbone network. Our experimental results show that our method is superior to state-of-the-art methods on three benchmark datasets (SMIC, CASME II, and SAMM) and a combined dataset (3DB).

Keywords: deep learning, optical flow, micro-expression recognition, multi-scale feature

References(42)

[1]
Haggard, E. A.; Isaacs, K. S. Micromomentary facial expressions as indicators of ego mechanisms in psychotherapy. In: Methods of Research in Psychotherapy. The Century Psychology Series. Boston: Springer, 154–165, 1966.
DOI
[2]
Ekman, P.; Friesen, W. V. Nonverbal leakage and clues to deception. Psychiatry Vol. 32, No. 1, 88–106, 1969.
DOI Google Scholar
[3]
Ekman, P. METT: Micro expression training tool. CD-ROM. Oakland, 2003.
[4]
Huang, X. H.; Zhao, G. Y.; Hong, X. P.; Zheng, W. M.; Pietikäinen, M. Spontaneous facial micro-expression analysis using Spatiotemporal Completed Local Quantized Patterns. Neurocomputing Vol. 175, 564–578, 2016.
DOI Google Scholar
[5]
Polikovsky, S.; Kameda, Y.; Ohta, Y. Facial micro-expressions recognition using high speed camera and 3D-gradient descriptor. In: Proceedings of the 3rd International Conference on Imaging for Crime Detection and Prevention, 1–6, 2009.
DOI
[6]
Zhao, G. Y.; Pietikainen, M. Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 29, No. 6, 915–928, 2007.
DOI Google Scholar
[7]
Liong, S. T.; See, J.; Wong, K.; Phan, R. C. W. Less is more: Micro-expression recognition from video using apex frame. Signal Processing: Image Communication Vol. 62, 82–92, 2018.
DOI Google Scholar
[8]
Liu, Y.-J.; Zhang, J.-K.; Yan, W.-J.; Wang, S.-J.; Zhao, G.; Fu, X. A main directional mean optical ow feature for spontaneous microexpression recognition. IEEE Transactions on Affective Computing Vol. 7, No. 4, 299–310, 2015.
DOI Google Scholar
[9]
Xu, F.; Zhang, J. P.; Wang, J. Z. Microexpression identification and categorization using a facial dynamics map. IEEE Transactions on Affective Computing Vol. 8, No. 2, 254–267, 2017.
DOI Google Scholar
[10]
Khor, H. Q.; See, J.; Phan, R. C. W.; Lin, W. Y. Enriched long-term recurrent convolutional network for facial micro-expression recognition. In: Proceedings of the 13th IEEE International Conference on Automatic Face & Gesture Recognition, 667–674, 2018.
DOI
[11]
Xia, Z. Q.; Hong, X. P.; Gao, X. Y.; Feng, X. Y.; Zhao, G. Y. Spatiotemporal recurrent convolutional networks for recognizing spontaneous micro-expressions. IEEE Transactions on Multimedia Vol. 22, No. 3, 626–640, 2019.
DOI Google Scholar
[12]
Gan, Y. S.; Liong, S. T.; Yau, W. C.; Huang, Y. C.; Tan, L. K. OFF-ApexNet on micro-expression recognition system. Signal Processing: Image Communication Vol. 74, 129–139, 2019.
DOI Google Scholar
[13]
Khor, H.-Q.; See, J.; Liong, S.-T.; Phan, R. C.; Lin, W. Dual-stream shallow networks for facial microexpression recognition. In: Proceedings of the IEEE International Conference on Image Processing, 36–40, 2019.
DOI
[14]
Shreve, M.; Godavarthy, S.; Manohar, V.; Goldgof, D.; Sarkar, S. Towards macro- and micro-expression spotting in video using strain patterns. In: Proceedings of the Workshop on Applications of Computer Vision, 1–6, 2009.
DOI
[15]
Li, X.; Pfister, T.; Huang, X.; Zhao, G.; Pietikäinen, M. A spontaneous micro-expression database: Inducement, collection and baseline. In: Proceedings of the 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, 1–6, 2013.
DOI
[16]
Yan, W. J.; Qi, W.; Liu, Y. J.; Wang, S. J.; Fu, X. L. CASME database: A dataset of spontaneous micro-expressions collected from neutralized faces. In: Proceedings of the 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, 1–7, 2013.
[17]
Yan, W. J.; Li, X.; Wang, S. J.; Zhao, G.; Liu, Y. J.; Chen, Y. H.; Fu, X. CASME II: An improved spontaneous micro-expression database and the baseline evaluation. PLoS One Vol. 9, No. 1, e86041, 2014.
DOI Google Scholar
[18]
Davison, A. K.; Lansley, C.; Costen, N.; Tan, K.; Yap, M. H. SAMM: A spontaneous micro-facial movement dataset. IEEE Transactions on Affective Computing Vol. 9, No. 1, 116–129, 2016.
DOI Google Scholar
[19]
Huang, X.; Zhao, G.; Hong, X.; Pietikäinen, M.; Zheng, W. Texture description with completed local quantized patterns. In: Image Analysis. Lecture Notes in Computer Science, Vol. 7944. Kämäräinen, J. K.; Koskela, M. Eds. Springer Berlin Heidelberg, 1–10, 2013.
DOI
[20]
Wang, Y.; See, J.; Phan, R. C. W.; Oh, Y. H. LBP with six intersection points: Reducing redundant information in LBP-TOP for micro-expression recognition. In: Computer Vision – ACCV 2014. Lecture Notes in Computer Science, Vol. 9003. Cremers, D.; Reid, I.; Saito, H.; Yang, M. H. Eds. Springer Cham, 525–537, 2015.
DOI
[21]
Ben, X. Y.; Jia, X. T.; Yan, R.; Zhang, X.; Meng, W. X. Learning effective binary descriptors for micro-expression recognition transferred by macro-information. Pattern Recognition Letters Vol. 107, 50–58, 2018.
DOI Google Scholar
[22]
Wang, S.-J.; Yan, W.-J.; Li, X.; Zhao, G.; Fu, X. Micro-expression recognition using dynamic textures on tensor independent color space. In: Proceedings of the 22nd International Conference on Pattern Recognition, 4678–4683, 2014.
DOI
[23]
Huang, X.; Wang, S.-J.; Liu, X.; Zhao, G.; Feng, X.; Pietikäinen, M. Discriminative spatiotemporal local binary pattern with revisited integral projection for spontaneous facial micro-expression recognition. IEEE Transactions on Affective Computing Vol. 10, No. 1, 32–47, 2017.
DOI Google Scholar
[24]
Sun, D.; Roth, S.; Black, M. J. A quantitative analysis of current practices in optical ow estimation and the principles behind them.International Journal of Computer Vision Vol. 106, No. 2, 115–137, 2014.
DOI Google Scholar
[25]
Zach, C.; Pock, T.; Bischof, H. A duality based approach for realtime TV-L1 optical flow. In: Pattern Recognition. Lecture Notes in Computer Science, Vol. 4713. Hamprecht, F. A.; Schnörr, C.; Jähne, B. Eds. Springer Berlin Heidelberg, 214–223, 2007.
[26]
Liu, Y. J.; Li, B. J.; Lai, Y. K. Sparse MDMO: Learning a discriminative feature for micro-expression recognition. IEEE Transactions on Affective Computing Vol. 12, No. 1, 254–261, 2021.
Google Scholar
[27]
Peng, S.; Huang, H. B.; Chen, W. J.; Zhang, L.; Fang, W. W. More trainable inception-ResNet for face recognition. Neurocomputing Vol. 411, 9–19, 2020.
DOI Google Scholar
[28]
Wang, S.; Cheng, Z.; Deng, X.; Chang, L.; Duan, F.; Lu, K. Leveraging 3D blendshape for facial expression recognition using CNN. Science China Information Sciences Vol. 63, No. 2, 120114, 2020.
DOI Google Scholar
[29]
Kim, D. H.; Baddar, W. J.; Ro, Y. M. Micro-expression recognition with expression-state constrained spatio-temporal feature representations. In: Proceedings of the 24th ACM international Conference on Multimedia, 382–386, 2016.
DOI
[30]
Peng, M.; Wu, Z.; Zhang, Z.; Chen, T. From macro to micro expression recognition: Deep learning on small datasets using transfer learning. In: Proceedings of the 13th IEEE International Conference on Automatic Face & Gesture Recognition, 657–661, 2018.
DOI
[31]
Wang, C. Y.; Peng, M.; Bi, T.; Chen, T. Micro-attention for micro-expression recognition. Neurocomputing Vol. 410, 354–362, 2020.
DOI Google Scholar
[32]
Wang, S. J.; Li, B. J.; Liu, Y. J.; Yan, W. J.; Ou, X. Y.; Huang, X. H.; Fu, X. Micro-expression recognition with small sample size by transferring long-term convolutional neural network. Neurocomputing Vol. 312, 251–262, 2018.
DOI Google Scholar
[33]
Wu, H. Y.; Rubinstein, M.; Shih, E.; Guttag, J.; Durand, F.; Freeman, W. Eulerian video magnification for revealing subtle changes in the world.ACM Transactions on Graphics Vol. 31, No. 4, Article No. 65, 2012.
DOI Google Scholar
[34]
Quang, N. V.; Chun, J.; Tokuyama, T. CapsuleNet for micro-expression recognition. In: Proceedings of the 14th IEEE International Conference on Automatic Face & Gesture Recognition, 1–7, 2019.
DOI
[35]
Peng, M.; Wang, C.; Bi, T.; Shi, Y.; Zhou, X.; Chen, T. A novel apex-time network for cross-dataset micro-expression recognition. In: Proceedings of the 8th International Conference on Affective Computing and Intelligent Interaction, 1–6, 2019.
DOI
[36]
Liu, Y. C.; Du, H. M.; Zheng, L.; Gedeon, T. A neural micro-expression recognizer. In: Proceedings of the 14th IEEE International Conference on Automatic Face & Gesture Recognition, 1–4, 2019.
DOI
[37]
Zhou, L.; Mao, Q. R.; Xue, L. Y. Dual-inception network for cross-database micro-expression recognition. In: Proceedings of the 14th IEEE International Conference on Automatic Face & Gesture Recognition, 1–5, 2019.
DOI
[38]
Liong, S. T.; Gan, Y. S.; See, J.; Khor, H. Q.; Huang, Y. C. Shallow triple stream three-dimensional CNN (STSTNet) for micro-expression recognition. In: Proceedings of the 14th IEEE International Conference on Automatic Face & Gesture Recognition, 1–5, 2019.
DOI
[39]
Li, Y. T.; Huang, X. H.; Zhao, G. Y. Can micro-expression be recognized based on single apex frame? In: Proceedings of the 25th IEEE International Conference on Image Processing, 3094–3098, 2018.
[40]
See, J.; Yap, M. H.; Li, J.; Hong, X.; Wang, S.-J. MEGC 2019 – The second facial micro-expressions grand challenge. In: Proceedings of the 14th IEEE International Conference on Automatic Face & Gesture Recognition, 1–5, 2019.
DOI
[41]
Krizhevsky, A.; Sutskever, I.; Hinton, G. E. ImageNet classification with deep convolutional neural networks. Communications of the ACM Vol. 60, No. 6, 84–90, 2017.
DOI Google Scholar
[42]
Le Ngo, A. C.; Phan, R. C. W.; See, J. Spontaneous subtle expression recognition: Imbalanced databases and solutions. In: Computer Vision – ACCV 2014. Lecture Notes in Computer Science, Vol. 9006. Cremers, D.; Reid, I.; Saito, H.; Yang, M. H. Eds. Springer Cham, 33–48, 2015.
DOI
Publication history
Copyright
Acknowledgements
Rights and permissions

Publication history

Received: 25 January 2021
Accepted: 25 February 2021
Published: 16 April 2021
Issue date: September 2021

Copyright

© The Author(s) 2021

Acknowledgements

The work was supported by the NSFC–Zhejiang Joint Fund of the Integration of Informatization and Industrialization under Grant No. U1909210, the the National Natural Science Foundation of China under Grant No. 61772312, and the Fundamental Research Funds of Shandong University (Grant No. 2018JC030).

Rights and permissions

This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduc-tion in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www. editorialmanager.com/cvmj.

Return