Journal Home > Volume 6 , Issue 1
Computational Visual Media 2020, 6(1): 95-112 https://doi.org/10.1007/s41095-020-0163-y
Research Article | Open Access | Issue | Published: 23 March 2020

Learning local shape descriptors for computing non-rigid dense correspondence

Show Author's Information Jianwei Guo1,  Hanyu Wang2,  Zhanglin Cheng3( ),  Xiaopeng Zhang1,  Dong-Ming Yan1( )
National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China.
University of Maryland-College Park, Maryland, USA.
Shenzhen VisuCA Key Lab, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China.
Keywords:
local feature descriptor, triplet CNN, dense correspondence, geometry image, non-rigid shape
Cite this article:
Guo J, Wang H, Cheng Z, et al. Learning local shape descriptors for computing non-rigid dense correspondence. Computational Visual Media, 2020, 6(1): 95-112. https://doi.org/10.1007/s41095-020-0163-y
Download citation
EndNote(RIS)
BibTeX

594

Views

25

Downloads

Citations

13

Crossref

N/A

WoS

13

Scopus

1

CSCD

Abstract Full text About this article

A discriminative local shape descriptor plays an important role in various applications. In this paper, we present a novel deep learning framework that derives discriminative local descriptors for deformable 3D shapes. We use local "geometry images" to encode the multi-scale local features of a point, via an intrinsic parameterization method based on geodesic polar coordinates. This new parameterization provides robust geometry images even for badly-shaped triangular meshes. Then a triplet network with shared architecture and parameters is used to perform deep metric learning; its aim is to distinguish between similar and dissimilar pairs of points. Additionally, a newly designed triplet loss function is minimized for improved, accurate training of the triplet network. To solve the dense correspondence problem, an efficient sampling approach is utilized to achieve a good compromise between training performance and descriptor quality. During testing, given a geometry image of a point of interest, our network outputs a discriminative local descriptor for it. Extensive testing of non-rigid dense shape matching on a variety of benchmarks demonstrates the superiority of the proposed descriptors over the state-of-the-art alternatives.


menu
Abstract
Full text
Outline
About this article

Learning local shape descriptors for computing non-rigid dense correspondence

Show Author's information Jianwei Guo1, Hanyu Wang2, Zhanglin Cheng3( ), Xiaopeng Zhang1, Dong-Ming Yan1( )
National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China.
University of Maryland-College Park, Maryland, USA.
Shenzhen VisuCA Key Lab, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China.

Abstract

A discriminative local shape descriptor plays an important role in various applications. In this paper, we present a novel deep learning framework that derives discriminative local descriptors for deformable 3D shapes. We use local "geometry images" to encode the multi-scale local features of a point, via an intrinsic parameterization method based on geodesic polar coordinates. This new parameterization provides robust geometry images even for badly-shaped triangular meshes. Then a triplet network with shared architecture and parameters is used to perform deep metric learning; its aim is to distinguish between similar and dissimilar pairs of points. Additionally, a newly designed triplet loss function is minimized for improved, accurate training of the triplet network. To solve the dense correspondence problem, an efficient sampling approach is utilized to achieve a good compromise between training performance and descriptor quality. During testing, given a geometry image of a point of interest, our network outputs a discriminative local descriptor for it. Extensive testing of non-rigid dense shape matching on a variety of benchmarks demonstrates the superiority of the proposed descriptors over the state-of-the-art alternatives.

Keywords: local feature descriptor, triplet CNN, dense correspondence, geometry image, non-rigid shape

References(66)

[1]
É. Corman,; M. Ovsjanikov,; A. Chambolle, Supervised descriptor learning for non-rigid shape matching. In: Computer Vision - ECCV 2014 Workshops. Lecture Notes in Computer Science, Vol. 8928. L. Agapito,; M. Bronstein,; C. Rother, Eds. Springer Cham, 283-298, 2015.
DOI
[2]
Y. L. Guo,; M. Bennamoun,; F. Sohel,; M. Lu,; J. W. Wan, 3D object recognition in cluttered scenes with local surface features: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 36, No. 11, 2270-2287, 2014.
DOI Google Scholar
[3]
Z. H. Lian,; A. Godil,; B. Bustos,; M. Daoudi,; J. Hermans,; S. Kawamura,; Y. Kurita,; G. Lavoué,; H. Van Nguyen,; R. Ohbuchi,; et al. A comparison of methods for non-rigid 3D shape retrieval. Pattern Recognition Vol. 46, No. 1, 449-461, 2013.
DOI Google Scholar
[4]
O. Van Kaick,; H. Zhang,; G. Hamarneh,; D. Cohen-Or, A survey on shape correspondence. Computer Graphics Forum Vol. 30, No. 6, 1681-1707, 2011.
DOI Google Scholar
[5]
Y. Q. Wang,; J. W. Guo,; D. M. Yan,; K. Wang,; X. P. Zhang, A robust local spectral descriptor for matching non-rigid shapes with incompatible shape structures. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 6231-6240, 2019.
DOI
[6]
S. A. A. Shah,; M. Bennamoun,; F. Boussaid, A novel 3D vorticity based approach for automatic registration of low resolution range images. Pattern Recognition Vol. 48, No. 9, 2859-2871, 2015.
DOI Google Scholar
[7]
A. E. Johnson,; M. Hebert, Using spin images for efficient object recognition in cluttered 3D scenes. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 21, No. 5, 433-449, 1999.
DOI Google Scholar
[8]
R. Gal,; D. Cohen-Or, Salient geometric features for partial shape matching and similarity. ACM Transactions on Graphics Vol. 25, No. 1, 130-150, 2006.
DOI Google Scholar
[9]
J. Sun,; M. Ovsjanikov,; L. Guibas, A concise and provably informative multi-scale signature based on heat diffusion. Computer Graphics Forum Vol. 28, No. 5, 1383-1392, 2009.
DOI Google Scholar
[10]
H. B. Huang,; E. Kalogerakis,; S. Chaudhuri,; D. Ceylan,; V. G. Kim,; E. Yumer, Learning local shape descriptors from part correspondences with multiview convolutional networks. ACM Transactions on Graphics Vol. 37, No. 1, Article No. 6, 2018.
DOI Google Scholar
[11]
A. Zeng,; S. R. Song,; M. NieBner,; M. Fisher,; J. X. Xiao,; T. Funkhouser, 3DMatch: Learning local geometric descriptors from RGB-D reconstructions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 199-208, 2017.
DOI
[12]
F. Monti,; D. Boscaini,; J. Masci,; E. Rodola,; J. Svoboda,; M. M. Bronstein, Geometric deep learning on graphs and manifolds using mixture model CNNs. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 5425-5434, 2017.
DOI
[13]
D. Anguelov,; P. Srinivasan,; D. Koller,; S. Thrun,; J. Rodgers,; J. Davis, SCAPE: Shape completion and animation of people. In: Proceedings of the SIGGRAPH ’05: ACM SIGGRAPH 2005 Papers, 408-416, 2005.
DOI
[14]
A. Bronstein,; M. Bronstein,; R. Kimmel, In the rigid kingdom. In: Numerical Geometry of Non-Rigid Shapes. Springer New York, 119-135, 2008.
DOI
[15]
A. Sinha,; J. Bai,; K. Ramani, Deep learning 3D shape surfaces using geometry images. In: Computer Vision - ECCV 2016. Lecture Notes in Computer Science, Vol. 9910. B. Leibe,; J. Matas,; N. Sebe,; M. Welling, Eds. Springer Cham, 223-240, 2016.
[16]
J. Wang,; Y. Song,; T. Leung,; C. Rosenberg,; J. B. Wang,; J. Philbin,; B Chen,; Y. Wu, Learning fine-grained image similarity with deep ranking. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1386-1393, 2014.
DOI
[17]
H. Y. Wang,; J. W. Guo,; D. M. Yan,; W. Z. Quan,; X. P. Zhang, Learning 3D keypoint descriptors for non-rigid shape matching. In: Computer Vision - ECCV 2018. Lecture Notes in Computer Science, Vol. 11212. V. Ferrari,; M. Hebert,; C. Sminchisescu,; Y. Weiss, Eds. Springer Cham, 3-20, 2018.
[18]
Y. L. Guo,; M. Bennamoun,; F. Sohel,; M. Lu,; J. W. Wan,; N. M. Kwok, A comprehensive performance evaluation of 3D local feature descriptors. International Journal of Computer Vision Vol. 116, No. 1, 66-89, 2016.
DOI Google Scholar
[19]
A. Frome,; D. Huber,; R. Kolluri,; T. Bülow,; J. Malik, Recognizing objects in range data using regional point descriptors. In: Computer Vision - ECCV 2004. Lecture Notes in Computer Science, Vol. 3023. T. Pajdla,; J. Matas, Eds. Springer Berlin Heidelberg, 224-237, 2004.
DOI
[20]
A. Zaharescu,; E. Boyer,; K. Varanasi,; R. Horaud, Surface feature detection and description with applications to mesh matching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 373-380, 2009.
DOI
[21]
F. Tombari,; S. Salti,; L. di Stefano, Unique signatures of histograms for local surface description. In: Computer Vision - ECCV 2010. Lecture Notes in Computer Science, Vol. 6313. K. Daniilidis; P. Maragos; N. Paragios Eds. Springer Berlin Heidelberg, 356-369, 2010.
DOI
[22]
A. M. Bronstein,; M. M. Bronstein,; L. J. Guibas,; M. Ovsjanikov, Shape google: Geometric words and expressions for invariant shape retrieval. ACM Transactions on Graphics Vol. 3, No. 1, Article No. 1, 2011.
DOI Google Scholar
[23]
Y. L. Guo,; F. Sohel,; M. Bennamoun,; M. Lu,; J. W. Wan, Rotational projection statistics for 3D local surface description and object recognition. International Journal of Computer Vision Vol. 105, No. 1, 63-86, 2013.
DOI Google Scholar
[24]
A. Elad,; R. Kimmel, On bending invariant signatures for surfaces. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 25, No. 10, 1285-1295, 2003.
DOI Google Scholar
[25]
M. Aubry,; U. Schlickewei,; D. Cremers, The wave kernel signature: A quantum mechanical approach to shape analysis. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, 1626-1633, 2011.
DOI
[26]
I. Kokkinos,; M. M. Bronstein,; R. Litman,; A. M. Bronstein, Intrinsic shape context descriptors for deformable shapes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 159-166, 2012.
DOI
[27]
R. Litman,; A. M. Bronstein, Learning spectral descriptors for deformable shape correspondence. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 36, No. 1, 171-180, 2014.
DOI Google Scholar
[28]
L. Gao,; Y. P. Cao,; Y. K. Lai,; H. Z. Huang,; L. Kobbelt,; S. M. Hu, Active exploration of large 3D model repositories. IEEE Transactions on Visualization and Computer Graphics Vol. 21, No. 12, 1390-1402, 2015.
DOI Google Scholar
[29]
Q. X. Huang,; G. X. Zhang,; L. Gao,; S. M. Hu,; A. Butscher,; L. Guibas, An optimization approach for extracting and encoding consistent maps in a shape collection. ACM Transactions on Graphics Vol. 31, No. 6, Article No. 167, 2012.
DOI Google Scholar
[30]
L. Y. Wei,; Q. X. Huang,; D. Ceylan,; E. Vouga,; H. Li, Dense human body correspondences using convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1544-1553, 2016.
DOI
[31]
R. Q. Charles,; S. Hao,; K. C. Mo,; L. J. Guibas, PointNet: Deep learning on point sets for 3D classification and segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 77-85, 2017.
DOI
[32]
M. Khoury,; Q.Y. Zhou,; V. Koltun, Learning compact geometric features. In: Proceedings of the IEEE International Conference on Computer Vision, 153-161, 2017.
DOI
[33]
S. Bai,; X. Bai,; Z. C. Zhou,; Z. X. Zhang,; Q. Tian,; L. J. Latecki, GIFT: Towards scalable 3D shape retrieval. IEEE Transactions on Multimedia Vol. 19, No. 6, 1257-1271, 2017.
DOI Google Scholar
[34]
M. M. Bronstein,; J. Bruna,; Y. LeCun,; A. Szlam,; P. Vandergheynst, Geometric deep learning: Going beyond Euclidean data. IEEE Signal Processing Magazine Vol. 34, No. 4, 18-42, 2017.
DOI Google Scholar
[35]
D. Boscaini,; J. Masci,; S. Melzi,; M. M. Bronstein,; U. Castellani,; P. Vandergheynst, Learning class-specific descriptors for deformable shapes using localized spectral convolutional networks. Computer Graphics Forum Vol. 34, No. 5, 13-23, 2015.
DOI Google Scholar
[36]
J. Masci,; D. Boscaini,; M. M. Bronstein,; P. Vandergheynst, Geodesic convolutional neural networks on Riemannian manifolds. In: Proceedings of the IEEE International Conference on Computer Vision Workshop, 37-45, 2015.
DOI
[37]
D. Boscaini,; J. Masci,; E. Rodolà,; M. Bronstein, Learning shape correspondence with anisotropic convolutional neural networks. In: Proceedings of the Advances in Neural Information Processing Systems, 3189-3197, 2016.
[38]
O. Litany,; T. Remez,; E. Rodola,; A. Bronstein,; M. Bronstein, Deep functional maps: Structured prediction for dense shape correspondence. In: Proceedings of the IEEE International Conference on Computer Vision, 5660-5668, 2017.
DOI
[39]
S. Biasotti,; A. Cerri,; A. Bronstein,; M. Bronstein, Recent trends, applications, and perspectives in 3D shape similarity assessment. Computer Graphics Forum Vol. 35, No. 6, 87-119, 2016.
DOI Google Scholar
[40]
M. Ovsjanikov,; E. Corman,; M. Bronstein,; E. Rodolà,; M. Ben-Chen,; L. Guibas,; F. Chazal,; A. Bronstein, Computing and processing correspondences with functional maps. In: Proceedings of the SIGGRAPH ASIA 2016 Courses, Article No. 9, 2016.
DOI
[41]
M. Ovsjanikov,; Q. Mérigot,; F. Mémoli,; L. Guibas, One point isometric matching with the heat kernel. Computer Graphics Forum Vol. 29, No. 5, 1555-1564, 2010.
DOI Google Scholar
[42]
F. Mémoli,; G. Sapiro, A theoretical and com-putational framework for isometry invariant recognition of point cloud data. Foundations of Computational Mathematics Vol. 5, No. 3, 313-347, 2005.
DOI Google Scholar
[43]
Q. F. Chen,; V. Koltun, Robust nonrigid registration by convex optimization. In: Proceedings of the IEEE International Conference on Computer Vision, 2039-2047, 2015.
DOI
[44]
M. Vestner,; R. Litman,; E. Rodola,; A. Bronstein,; D. Cremers, Product manifold filter: Non-rigid shape correspondence via kernel density estimation in the product space. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 6681-6690, 2017.
DOI
[45]
R. R. Coifman,; S. Lafon,; A. B. Lee,; M. Maggioni,; B. Nadler,; F. Warner,; S. W. Zucker, Geometric diffusions as a tool for harmonic analysis and structure definition of data: Diffusion maps. Proceedings of the National Academy of Sciences of the United States of America Vol. 102, No. 21, 7426-7431, 2005.
DOI Google Scholar
[46]
M. Ovsjanikov,; M. Ben-Chen,; J. Solomon,; A. Butscher,; L. Guibas, Functional maps: A exible representation of maps between shapes. ACM Transactions on Graphics Vol. 31, No. 4, Article No. 30, 2012.
DOI Google Scholar
[47]
J. Pokrass,; A. M. Bronstein,; M. M. Bronstein,; P. Sprechmann,; G. Sapiro, Sparse modeling of intrinsic correspondences. Computer Graphics Forum Vol. 32, No. 2pt4, 459-468, 2013.
DOI Google Scholar
[48]
A. Kovnatsky,; M. M. Bronstein,; X. Bresson,; P. Vandergheynst, Functional correspondence by matrix completion. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 905-914, 2015.
DOI
[49]
D. Nogneng,; M. Ovsjanikov, Informative descriptor preservation via commutativity for shape matching. Computer Graphics Forum Vol. 36, No. 2, 259-267, 2017.
DOI Google Scholar
[50]
E. Rodola,; S. Rota Bulo,; T. Windheuser,; M. Vestner,; D. Cremers, Dense non-rigid shape correspondence using random forests. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4177-4184, 2014.
DOI
[51]
X. F. Gu,; S. J. Gortler,; H. Hoppe, Geometry images. ACM Transactions on Graphics Vol. 21, No. 3, 355-361, 2002.
DOI Google Scholar
[52]
F. Bogo,; J. Romero,; M. Loper,; M. J. Black, FAUST: Dataset and evaluation for 3D mesh registration. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3794-3801, 2014.
DOI
[53]
I. Sipiran,; B. Bustos, Harris 3D: A robust extension of the Harris operator for interest point detection on 3D meshes. The Visual Computer Vol. 27, No. 11, 963-976, 2011.
DOI Google Scholar
[54]
D. M. Yan,; J. W. Guo,; X. H. Jia,; X. P. Zhang,; P. Wonka, Blue-noise remeshing with farthest point optimization. Computer Graphics Forum Vol. 33, No. 5, 167-176, 2014.
DOI Google Scholar
[55]
D. Boscaini,; J. Masci,; E. Rodolà,; M. M. Bronstein,; D. Cremers, Anisotropic diffusion descriptors. Computer Graphics Forum Vol. 35, No. 2, 431-441, 2016.
DOI Google Scholar
[56]
E. L. Melvaer,; M. Reimers, Geodesic polar coordinates on polygonal meshes. Computer Graphics Forum Vol. 31, No. 8, 2423-2435, 2012.
DOI Google Scholar
[57]
F. Schroff,; D. Kalenichenko,; J. Philbin, FaceNet: A unified embedding for face recognition and clustering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 815-823, 2015.
DOI
[58]
B, G. Vijay Kumar,; G. Carneiro,; I. Reid, Learning local image descriptors with deep Siamese and triplet convolutional networks by minimizing global loss functions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 5385-5394, 2016.
DOI
[59]
J. Svoboda,; J. Masci,; M. M. Bronstein, Palmprint recognition via discriminative index learning. In: Proceedings of the 23rd International Conference on Pattern Recognition, 4232-4237, 2016.
DOI
[60]
S. Ioffe,; C. Szegedy, Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: Proceedings of the 32nd International Conference on Machine Learning, 448-456, 2015.
[61]
A. L. Maas,; A. Y. Hannun,; A. Y. Ng, Rectifier nonlinearities improve neural network acoustic models. In: Proceedings of the 30th International Conference on Machine Learning, 2013.
[62]
X. Glorot,; Y. Bengio, Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the 13th International Conference on Artificial Intelligence and Statistics, 249-256, 2010.
[63]
D. Kingma,; J. Ba, Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[64]
Y. P. Yang,; Y. Yu,; Y. Zhou,; S. D. Du,; J. Davis,; R. G. Yang, Semantic parametric reshaping of human body models. In: Proceedings of the 2nd International Conference on 3D Vision, 41-48, 2014.
DOI
[65]
M. Abadi,; A. Agarwal,; P. Barham,; E. Brevdo,; Z. Chen,; C. Citro,; G. S. Corrado,; A. Davis,; J. Dean,; M.; Devin, et al. TensorFlow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467, 2016.
[66]
L. Cosmo,; E. Rodolà,; M. M. Bronstein,; A. Torsello,; D. Cremers,; Y. Sahillioglu, SHREC’16: Partial matching of deformable shapes. In: Proceedings of the Eurographics Workshop on 3D Object Retrieval, 2016.
Publication history
Copyright
Acknowledgements
Rights and permissions

Publication history

Revised: 22 January 2020
Accepted: 29 February 2020
Published: 23 March 2020
Issue date: March 2020

Copyright

© The author(s) 2020

Acknowledgements

This work was partially funded by the National Key R&D Program of China (2018YFB2100602), the National Natural Science Foundation of China (61802406, 61772523, 61702488), Beijing Natural Science Foundation (L182059), the CCF-Tencent Open Research Fund, Shenzhen Basic Research Program (JCYJ20180507182222355), and the Open Project Program of the State Key Lab of CAD&CG (A2004) Zhejiang University.

Rights and permissions

This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduc-tion in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www. editorialmanager.com/cvmj.

Return