Journal Home > Volume 7 , Issue 2
Computational Visual Media 2021, 7(2): 159-167 https://doi.org/10.1007/s41095-020-0189-1
Review Article | Open Access | Issue | Published: 23 October 2020

Machine learning for digital try-on: Challenges and progress

Show Author's Information Junbang Liang1( ),  Ming C. Lin1
University of Maryland, College Park, MD 20785, USA
Keywords:
machine learning, digital try-on, garmentmodeling, human body estimation, material modeling
Cite this article:
Liang J, Lin MC. Machine learning for digital try-on: Challenges and progress. Computational Visual Media, 2021, 7(2): 159-167. https://doi.org/10.1007/s41095-020-0189-1
Download citation
EndNote(RIS)
BibTeX

610

Views

27

Downloads

Citations

6

Crossref

N/A

WoS

8

Scopus

2

CSCD

Abstract Full text About this article

Digital try-on systems for e-commerce have the potential to change people’s lives and provide notable economic benefits. However, their development is limited by practical constraints, such as accurate sizing of the body and realism of demonstrations. We enumerate three open challenges remaining for a complete and easy-to-use try-on system that recent advances in machine learningmake increasingly tractable. For each, we describethe problem, introduce state-of-the-art approaches, and provide future directions.


menu
Abstract
Full text
Outline
About this article

Machine learning for digital try-on: Challenges and progress

Show Author's information Junbang Liang1( ), Ming C. Lin1
University of Maryland, College Park, MD 20785, USA

Abstract

Digital try-on systems for e-commerce have the potential to change people’s lives and provide notable economic benefits. However, their development is limited by practical constraints, such as accurate sizing of the body and realism of demonstrations. We enumerate three open challenges remaining for a complete and easy-to-use try-on system that recent advances in machine learningmake increasingly tractable. For each, we describethe problem, introduce state-of-the-art approaches, and provide future directions.

Keywords: machine learning, digital try-on, garmentmodeling, human body estimation, material modeling

References(29)

[1]
Zheng, Z. H.; Zhang, H. T.; Zhang, F. L.; Mu, T. J. Image-based clothes changing system. Computational Visual Media Vol. 3, No. 4, 337-347, 2017.
DOI Google Scholar
[2]
Dibra, E.; Jain, H.; Öztireli, C.; Ziegler, R.; Gross, M. HS-Nets: Estimating human body shape from silhouettes with convolutional neural networks. In: Proceedings of the 4th International Conference on 3D Vision, 108-117, 2016.
DOI
[3]
Bălan, A. O.; Black, M. J. The naked truth: Estimating body shape under clothing. In: Computer Vision - ECCV 2008. Lecture Notes in Computer Science, Vol. 5303. Forsyth, D.; Torr, P.; Zisserman, A. Eds. Springer Berlin, 15-29, 2008.
[4]
Lassner, C.; Romero, J.; Kiefel, M.; Bogo, F.; Black, M. J.; Gehler, P. V. Unite the people: Closing the loop between 3D and 2D human representations. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4704-4713, 2017.
DOI
[5]
Loper, M.; Mahmood, N.; Romero, J.; Pons-Moll, G.; Black, M. J. SMPL: A skinned multi-person linear model. ACM Transactions on Graphics Vol. 34, No. 6, Article No. 248, 2015.
DOI Google Scholar
[6]
Wei, S.-E.; Ramakrishna, V.; Kanade, T.; Sheikh, Y. Convolutional pose machines. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 4724-4732, 2016.
DOI
[7]
Cao, Z.; Simon, T.; Wei, S.; Sheikh, Y. Realtime multi-person 2D pose estimation using part affinity fields. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1302-1310, 2017.
DOI
[8]
Mehta, D.; Sridhar, S.; Sotnychenko, O.; Rhodin, H.; Shafiei, M.; Seidel, H.-P.; Xu, W.; Casas, D.; Theobalt, C. VNect: Realtime 3D human pose estimation with a single RGB camera. ACM Transactions on Graphics Vol. 36, No. 4, Article No. 44, 2017.
DOI Google Scholar
[9]
Alldieck, T.; Magnor, M.; Xu, W.; Theobalt, C.; Pons-Moll, G. Video based reconstruction of 3D people models. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 8387-8397, 2018.
DOI
[10]
Kanazawa, A.; Black, M. J.; Jacobs, D. W.; Malik, J. End-to-end recovery of human shape and pose. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 7122-7131, 2018.
DOI
[11]
Varol, G.; Ceylan, D.; Russell, B.; Yang, J.; Yumer, E.; Laptev, I. Bodynet: Volumetric inference of 3D human body shapes. In: Proceedings of the European Conference on Computer Vision, 20-36, 2018.
DOI
[12]
Zheng, Z.; Yu, T.; Wei, Y.; Dai, Q.; Liu, Y. Deephuman: 3D human reconstruction from a single image. In: Proceedings of the IEEE International Conference on Computer Vision, 7739-7749, 2019.
DOI
[13]
Saito, S.; Huang, Z.; Natsume, R.; Morishima, S.; Kanazawa, A.; Li, H. PIFu: Pixel-aligned implicit function for high-resolution clothed human digitization. In: Proceedings of the IEEE International Conference on Computer Vision, 2304-2314, 2019.
DOI
[14]
Xu, Y.; Zhu, S.-C.; Tung, T. Denserac: Joint 3D pose and shape estimation by dense render-and-compare. In: Proceedings of the IEEE International Conference on Computer Vision, 7760-7770, 2019.
DOI
[15]
Smith, D.; Loper, M.; Hu, X.; Mavroidis, P.; Romero, J. FACSIMILE: Fast and accurate scans from an image in less than a second. In: Proceedings of the IEEE International Conference on Computer Vision, 5329-5338, 2019.
DOI
[16]
Alldieck, T.; Magnor, M.; Bhatnagar, B. L.; Theobalt, C.; Pons-Moll, G. Learning to reconstruct people in clothing from a single RGB camera. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1175-1186, 2019.
DOI
[17]
Kolotouros, N.; Pavlakos, G.; Black, M. J.; Daniilidis, K. Learning to reconstruct 3D human pose and shape via modelfitting in the loop. In: Proceedings of the IEEE International Conference on Computer Vision, 2252-2261, 2019.
DOI
[18]
Liang, J.; Lin, M. C. Shape-aware human pose and shape reconstruction using multi-view images. In: Proceedings of the IEEE International Conference on Computer Vision, 4352-4362, 2019.
DOI
[19]
Yang, S.; Pan, Z. R.; Amert, T.; Wang, K.; Yu, L. C.; Berg, T.; Lin, M. C. Physics-inspired garment recovery from a single-view image. ACM Transactions on Graphics Vol. 37, No. 5, Article No. 170, 2018.
DOI Google Scholar
[20]
Yang, S.; Liang, J.; Lin, M. C.; Learning-based cloth material recovery from video. In: Proceedings of the IEEE International Conference on Computer Vision, 4383-4393, 2017.
DOI
[21]
Qiao, Y. L.; Liang, J. B.; Koltun, V.; Lin, M. C. Scalable differentiable physics for learning and control. arXiv preprint arXiv:2007.02168, 2020.
Google Scholar
[22]
De Avila Belbute-Peres, F.; Smith, K. A.; Allen, K.; Tenenbaum, J.; Kolter, J. Z. End-to-end differentiable physics for learning and control. In: Proceedings of the Advances in Neural Information Processing Systems, 2018.
[23]
Degrave, J.; Hermans, M.; Dambre, J.; Wyffels, F. A differentiable physics engine for deep learning in robotics. Frontiers in Neurorobotics Vol. 13, 6, 2019.
DOI Google Scholar
[24]
Liang, J.; Lin, M.; Koltun, V. Differentiable cloth simulation for inverse problems. In: Proceedings of the 33rd Conference on Neural Information Processing Systems, 2019.
[25]
Hu, Y.; Liu, J.; Spielberg, A.; Tenenbaum, J. B.; Freeman, W. T.; Wu, J.; Rus, D.; Matusik, W. ChainQueen: A real-time differentiable physical simulator for soft robotics. In: Proceedings of the International Conference on Robotics and Automation, 6265-6271, 2019.
DOI
[26]
Hu, Y. M.; Anderson, L.; Li, T. M.; Sun, Q.; Carr, N.; Ragan-Kelley, J.; Durand, F. DiffTaichi: Differentiable programming for physical simulation. arXiv preprint arXiv:1910.00935, 2019.
Google Scholar
[27]
Liu, K. X.; Zeng, X. Y.; Bruniaux, P.; Tao, X. Y.; Yao, X. F.; Li, V.; Wang, J. 3D interactive garment pattern-making technology. Computer-Aided Design Vol. 104, 113-124, 2018.
DOI Google Scholar
[28]
Huang, P.; Yao, J.; Zhao, H. Automatic realistic 3D garment generation based on two images. In: Proceedings of the International Conference on Virtual Reality and Visualization, 250-257, 2016.
DOI
[29]
Wang, T. Y.; Ceylan, D.; Popović, J.; Mitra, N. J. Learning a shared shape space for multimodal garment design. ACM Transactions on Graphics Vol. 37, No. 6, Article No. 203, 2018.
DOI Google Scholar
Publication history
Copyright
Acknowledgements
Rights and permissions

Publication history

Received: 24 June 2020
Accepted: 21 July 2020
Published: 23 October 2020
Issue date: June 2021

Copyright

© The Author(s) 2020

Acknowledgements

This research was supported in part by the Iribe Professorship and the National Science Foundation.

Rights and permissions

This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduc-tion in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www. editorialmanager.com/cvmj.

Return