Journal Home > Volume 8 , Issue 2

There is a steadily growing range of applications that can benefit from facial reconstruction techniques, leading to an increasing demand for reconstruction of high-quality 3D face models. While it is an important expressive part of the human face, the nose has received less attention than other expressive regions in the face reconstruction literature. When applying existing reconstruction methods to facial images, the reconstructed nose models are often inconsistent with the desired shape and expression. In this paper, we propose a coarse-to-fine 3D nose reconstruction and correction pipeline to build a nosemodel from a single image, where 3D and 2D nose curve correspondences are adaptively updated and refined. We first correct the reconstruction result coarsely using constraints of 3D-2D sparse landmark correspondences, and then heuristically update a dense 3D-2D curve correspondence based on the coarsely corrected result. A final refinement step is performed to correct the shape based on the updated 3D-2D dense curve constraints. Experimental results show the advantages of our method for 3D nose reconstruction over existing methods.


menu
Abstract
Full text
Outline
About this article

3D corrective nose reconstruction from a single image

Show Author's information Yanlong Tang1Yun Zhang2( )Xiaoguang Han3Fang-Lue Zhang4Yu-Kun Lai5Ruofeng Tong6
Tencent Games Lightspeed & Quantum Studios, Shenzhen, China
Communication University of Zhejiang, Hangzhou, China
Shenzhen Research Institute of Big Data, the Chinese University of Hong Kong (Shenzhen), Shenzhen, China
Victoria University of Wellington, Wellington, New Zealand
Cardiff University, Wales, UK
Zhejiang University, Hangzhou, China

Abstract

There is a steadily growing range of applications that can benefit from facial reconstruction techniques, leading to an increasing demand for reconstruction of high-quality 3D face models. While it is an important expressive part of the human face, the nose has received less attention than other expressive regions in the face reconstruction literature. When applying existing reconstruction methods to facial images, the reconstructed nose models are often inconsistent with the desired shape and expression. In this paper, we propose a coarse-to-fine 3D nose reconstruction and correction pipeline to build a nosemodel from a single image, where 3D and 2D nose curve correspondences are adaptively updated and refined. We first correct the reconstruction result coarsely using constraints of 3D-2D sparse landmark correspondences, and then heuristically update a dense 3D-2D curve correspondence based on the coarsely corrected result. A final refinement step is performed to correct the shape based on the updated 3D-2D dense curve constraints. Experimental results show the advantages of our method for 3D nose reconstruction over existing methods.

Keywords: nose shape recovery, single image 3D reconstruction, contour correspondence, Laplacian deformation

References(27)

[1]
Samad, M. D.; Iftekharuddin, K. M. Frenet frame-based generalized space curve representation for pose-invariant classification and recognition of 3-D face. IEEE Transactions on Human-Machine Systems Vol. 46, No. 4, 522-533, 2016.
[2]
Werghi, N.; Tortorici, C.; Berretti, S.; Del Bimbo, A. Boosting 3D LBP-based face recognition by fusing shape and texture descriptors on the mesh. IEEE Transactions on Information Forensics and Security Vol. 11, No. 5, 964-979, 2016.
[3]
Li, H.; Yu, J. H.; Ye, Y. T.; Bregler, C. Realtime facial animation with on-the-fly correctives. ACM Transactions on Graphics Vol. 32, No. 4, Article No. 42, 2013.
[4]
Li, Y.; Ma, L. Q.; Fan, H. Q.; Mitchell, K. Feature-preserving detailed 3D face reconstruction from a single image. In: Proceedings of the 15th ACM SIGGRAPH European Conference on Visual Media Production, Article No. 1, 2018.
DOI
[5]
Jiang, L.; Zhang, J. Y.; Deng, B. L.; Li, H.; Liu, L. G. 3D face reconstruction with geometry details from a single image. IEEE Transactions on Image Processing Vol. 27, No. 10, 4756-4770, 2018.
[6]
Tang, Y. L.; Han, X. G.; Li, Y.; Ma, L. Q.; Tong, R. F. Expressive facial style transfer for personalized memes mimic. The Visual Computer Vol. 35, 783-795, 2019.
[7]
Kass, M.; Witkin, A.; Terzopoulos, D. Snakes: Active contour models. International Journal of Computer Vision Vol. 1, No. 4, 321-331, 1988.
[8]
Tang, Y. L.; Tong, R. F.; Tang, M.; Zhang, Y. Depth incorporating with color improves salient object detection. The Visual Computer Vol. 32, 111-121, 2016.
[9]
Blanz, V.; Vetter, T. A morphable model for the synthesis of 3D faces. In: Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, 187-194, 1999.
DOI
[10]
Paysan, P.; Knothe, R.; Amberg, B.; Romdhani, S.; Vetter, T. A 3D face model for pose and illumination invariant face recognition. In: Proceedings of the 6th IEEE International Conference on Advanced Video and Signal Based Surveillance, 296-301, 2009.
DOI
[11]
Booth, J.; Roussos, A.; Zafeiriou, S.; Ponniah, A.; Dunaway, D. A 3D morphable model learnt from 10,000 faces. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 5543-5552, 2016.
DOI
[12]
Zhu, X. Y.; Lei, Z.; Liu, X. M.; Shi, H. L.; Li, S. Z. Face alignment across large poses: A 3D solution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 146-155, 2016.
DOI
[13]
Cao, C.; Weng, Y. L.; Zhou, S.; Tong, Y. Y.; Zhou, K. FaceWarehouse: A 3D facial expression database for visual computing. IEEE Transactions on Visualization and Computer Graphics Vol. 20, No. 3, 413-425, 2014.
[14]
Feng, Y.; Wu, F.; Shao, X. H.; Wang, Y. F.; Zhou, X. Joint 3D face reconstruction and dense alignment with position map regression network. In: Computer Vision-ECCV 2018. Lecture Notes in Computer Science, Vol.11218. Ferrari, V.; Hebert, M.; Sminchisescu, C.; Weiss, Y. Eds. Springer Cham, 557-574, 2018.
DOI
[15]
Sanyal, S.; Bolkart, T.; Feng, H. W.; Black, M. J. Learning to regress 3D face shape and expression from an image without 3D supervision. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 7755-7764, 2019.
DOI
[16]
Deng, Y.; Yang, J. L.; Xu, S. C.; Chen, D.; Jia, Y. D.; Tong, X. Accurate 3D face reconstruction with weakly-supervised learning: From single image to image set. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 285-295, 2019.
DOI
[17]
Garrido, P.; Zollhöfer, M.; Casas, D.; Valgaerts, L.; Varanasi, K.; Pérez, P.; Theobalt, C. Reconstruction of personalized 3D face rigs from monocular video. ACM Transactions on Graphics Vol. 35, No. 3, Article No. 28, 2016.
[18]
Sorkine, O.; Cohen-Or, D.; Lipman, Y.; Alexa, M.; Rössl, C.; Seidel, H. P. Laplacian surface editing. In: Proceedings of the Eurographics/ACM SIGGRAPH symposium on Geometry processing, 175-184, 2004.
DOI
[19]
Bouaziz, S.; Wang, Y. G.; Pauly, M. Online modeling for realtime facial animation. ACM Transactions on Graphics Vol. 32, No. 4, Article No. 40, 2013.
[20]
Vallet, B.; Lévy, B. Spectral geometry processing with manifold harmonics. Computer Graphics Forum Vol. 27, No. 2, 251-260, 2008.
[21]
Wen, Q.; Xu, F.; Lu, M.; Yong, J. H. Real-time 3D eyelids tracking from semantic edges. ACM Transactions on Graphics Vol. 36, No. 6, Article No. 193, 2017.
[22]
Garrido, P.; Zollhöfer, M.; Wu, C. L.; Bradley, D.; Pérez, P.; Beeler, T.; Theobalt, C. Corrective 3D reconstruction of lips from monocular video. ACM Transactions on Graphics Vol. 35, No. 6, Article No. 219, 2016.
[23]
Dinev, D.; Beeler, T.; Bradley, D.; Bächer, M.; Xu, H.; Kavan, L. User-guided lip correction for facial performance capture. Computer Graphics Forum Vol. 37, No. 8, 93-101, 2018.
[24]
Feng, Z. H.; Huber, P.; Kittler, J.; Hancock, P.; Wu, X. J.; Zhao, Q. J.; Koppen, P.; Raetsch, M. Evaluation of dense 3D reconstruction from 2D face images in the wild. In: Proceedings of the 13th IEEE International Conference on Automatic Face & Gesture Recognition, 780-786, 2018.
DOI
[25]
Yin, L. J.; Wei, X. Z.; Sun, Y.; Wang, J.; Rosato, M. J. A 3D facial expression database for facial behavior research. In: Proceedings of the 7th International Conference on Automatic Face and Gesture Recognition, 211-216, 2006.
[26]
Thies, J.; Zollhöfer, M.; Stamminger, M.; Theobalt, C.; Nießner, M. Face2Face: Real-time face capture and reenactment of RGB videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2387-2395, 2016.
DOI
[27]
Canny, J. A computational approach to edge detection. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. PAMI-8, No. 6, 679-698, 1986.
Publication history
Copyright
Acknowledgements
Rights and permissions

Publication history

Received: 20 March 2021
Accepted: 29 April 2021
Published: 06 December 2021
Issue date: June 2022

Copyright

© The Author(s) 2021.

Acknowledgements

This research was supported by the National Natural Science Foundation of China (Grant Nos. 61972342, 61602402, and 61902334), Zhejiang Provincial Basic Public Welfare Research (Grant No. LGG19F020001), Shenzhen Fundamental Research (General Project) (Grant No. JCYJ20190814112007258), and the Royal Society (Grant No. IES\R1\180126).

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduc-tion in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www. editorialmanager.com/cvmj.

Return