Journal Home > Volume 3 , Issue 4
Computational Visual Media 2017, 3(4): 337-347 https://doi.org/10.1007/s41095-017-0084-6
Research Article | Open Access | Issue | Published: 08 May 2017

Image-based clothes changing system

Show Author's Information Zhao-Heng Zheng1,  Hao-Tian Zhang2,  Fang-Lue Zhang3( ),  Tai-Jiang Mu4
Computer Science and Engineering, University of Michigan, 2260 Hayward St, Ann Arbor, MI 48109, USA.
Computer Science Department, Stanford University, 353 Serra Mall, Stanford, CA 94305, USA.
School of Engineering and Computer Science, Victoria University of Wellington, Wellington, New Zealand.
TNList, Tsinghua University, Beijing 100084, China.
Keywords:
clothing try-on, image warping, human segmentation
Cite this article:
Zheng Z-H, Zhang H-T, Zhang F-L, et al. Image-based clothes changing system. Computational Visual Media, 2017, 3(4): 337-347. https://doi.org/10.1007/s41095-017-0084-6
Download citation
EndNote(RIS)
BibTeX

537

Views

15

Downloads

Citations

9

Crossref

N/A

WoS

11

Scopus

3

CSCD

Abstract Full text About this article

Current image-editing tools do not match up to the demands of personalized image manipulation, one application of which is changing clothes in user-captured images. Previous work can change single color clothes using parametric human warping methods. In this paper, we propose an image-based clothes changing system, exploiting body factor extraction and content-aware image warping. Image segmentation and mask generation are first applied to the user input. Afterwards, we determine joint positions via a neural network. Then, body shape matching is performed and the shape of the model is warped to the user’s shape. Finally, head swapping is performed to produce realistic virtual results. We also provide a supervision and labeling tool for refinement and further assistance when creating a dataset.


menu
Abstract
Full text
Outline
About this article

Image-based clothes changing system

Show Author's information Zhao-Heng Zheng1, Hao-Tian Zhang2, Fang-Lue Zhang3( ), Tai-Jiang Mu4
Computer Science and Engineering, University of Michigan, 2260 Hayward St, Ann Arbor, MI 48109, USA.
Computer Science Department, Stanford University, 353 Serra Mall, Stanford, CA 94305, USA.
School of Engineering and Computer Science, Victoria University of Wellington, Wellington, New Zealand.
TNList, Tsinghua University, Beijing 100084, China.

Abstract

Current image-editing tools do not match up to the demands of personalized image manipulation, one application of which is changing clothes in user-captured images. Previous work can change single color clothes using parametric human warping methods. In this paper, we propose an image-based clothes changing system, exploiting body factor extraction and content-aware image warping. Image segmentation and mask generation are first applied to the user input. Afterwards, we determine joint positions via a neural network. Then, body shape matching is performed and the shape of the model is warped to the user’s shape. Finally, head swapping is performed to produce realistic virtual results. We also provide a supervision and labeling tool for refinement and further assistance when creating a dataset.

Keywords: clothing try-on, image warping, human segmentation

References(35)

[1]
T. Chen,; P. Tan,; L. Q. Ma,; M. M. Cheng,; A. Shamir,; S.-M Hu,. PoseShop: Human image database construction and personalized content synthesis. IEEE Transactions on Visualization and Computer Graphics Vol. 19, No. 5, 824-837, 2013.
DOI Google Scholar
[2]
S. Zhou,; H. Fu,; L. Liu,; D. Cohen-Or,; X Han,. Parametric reshaping of human bodies in images. ACM Transactions on Graphics Vol. 29, No. 4, Article No. 126, 2010.
DOI Google Scholar
[3]
V. Ferrari,; M. Marin-Jimenez,; A. Zisserman, Progressive search space reduction for human pose estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1-8, 2008.
DOI
[4]
M. Andriluka,; S. Roth,; B. Schiele, Pictorial structures revisited: People detection and articulated pose estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1014-1021, 2009.
DOI
[5]
S. Johnson,; M. Everingham, Combining discriminative appearance and segmentation cues for articulated human pose estimation. In: Proceedings of the IEEE 12th International Conference on Computer Vision Workshops, 405-412, 2009.
DOI
[6]
M. Sun,; S. Savarese, Articulated part-based model for joint object detection and pose estimation. In: Proceedings of the International Conference on Computer Vision, 723-730, 2011.
DOI
[7]
Y. Tian,; C. L. Zitnick,; S. G. Narasimhan, Exploring the spatial hierarchy of mixture models for human pose estimation. In: Computer Vision-ECCV 2012. A. Fitzgibbon,; S. Lazebnik,; P. Perona,; Y. Sato,; C. Schmid, Eds. Springer-Verlag Berlin Heidelberg, 256-269, 2012.
[8]
Y. Yang,; D. Ramanan, Articulated pose estimation with flexible mixtures-of-parts. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1385-1392, 2011.
DOI
[9]
V. Vineet,; J. Warrell,; L. Ladicky,; P. H. S. Torr, Human instance segmentation from video using detector-based conditional random fields. In: Proceedings of the 22nd British Machine Vision Conference, 80.1-80.11, 2011.
DOI
[10]
J. J. Tompson,; A. Jain,; Y. LeCun,; C. Bregler, Joint training of a convolutional network and a graphical model for human pose estimation. In: Proceedings of the Advances in Neural Information Processing Systems 27, 1799-1807, 2014.
[11]
V. Ramakrishna,; D. Munoz,; M. Hebert,; J. A. Bagnell,; Y. Sheikh, Pose machines: Articulated pose estimation via inference machines. In: Computer Vision-ECCV 2014. D. Fleet,; T. Pajdla,; B. Schiele,; T. Tuytelaars, Eds. Springer International Publishing Switzerland, 33-47, 2014.
DOI
[12]
J. Carreira,; P. Agrawal,; K. Fragkiadaki,; J. Malik, Human pose estimation with iterative error feedback. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4733-4742, 2016.
DOI
[13]
S.-E. Wei,; V. Ramakrishna,; T. Kanade,; Y. Sheikh, Convolutional pose machines. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4724-4732, 2016.
DOI
[14]
A. Blake,; C. Rother,; M. Brown,; P. Perez,; P. Torr, Interactive image segmentation using an adaptive GMMRF model. In: Computer Vision-ECCV 2004. T. Pajdla,; J. Matas, Eds. Springer-Verlag Berlin Heidelberg, 428-441, 2004.
DOI
[15]
L.-C. Chen,; G. Papandreou,; I. Kokkinos,; K. Murphy,; A. L. Yuille, Semantic image segmentation with deep convolutional nets and fully connected CRFS. In: Proceedings of the International Conference on Learning Representations, 2015.
[16]
L. Ladický,; C. Russell,; P. Kohli,; P. H. S. Torr, Associative hierarchical CRFs for object class image segmentation. In: Proceedings of the IEEE 12th International Conference on Computer Vision, 739-746, 2009.
DOI
[17]
P. Arbeláez,; B. Hariharan,; C. Gu,; S. Gupta,; L. Bourdev,; J. Malik, Semantic segmentation using regions and parts. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3378-3385, 2012.
DOI
[18]
P. Arbeláez,; M. Maire,; C. Fowlkes,; J Malik,. Contour detection and hierarchical image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 33, No. 5, 898-916, 2011.
DOI Google Scholar
[19]
P. Arbeláez,; J. Pont-Tuset,; J. T. Barron,; F. Marques,; J. Malik, Multiscale combinatorial grouping. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 328-335, 2014.
DOI
[20]
B. C. Ko,; J.-Y Nam,. Object-of-interest image segmentation based on human attention and semantic region clustering. Journal of the Optical Society of America A Vol. 23, No. 10, 2462-2470, 2006.
DOI Google Scholar
[21]
M.-M. Cheng,; N. J. Mitra,; X. Huang,; P. H. S Torr,; S.-M Hu,. Global contrast based salient region detection. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 37, No. 3, 569-582, 2015.
DOI Google Scholar
[22]
M.-M. Cheng,; J. Warrell,; W.-Y. Lin,; S. Zheng,; V. Vineet,; N. Crook, Efficient salient region detection with soft image abstraction. In: Proceedings of the IEEE International Conference on Computer Vision, 1529-1536, 2013.
DOI
[23]
L.-C. Chen,; Y. Yang,; J. Wang,; W. Xu,; A. L. Yuille, Attention to scale: Scale-aware semantic image segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3640-3649, 2016.
DOI
[24]
S. Zheng,; M.-M. Cheng,; J. Warrell,; P. Sturgess,; V. Vineet,; C. Rother,; P. H. S. Torr, Dense semantic image segmentation with objects and attributes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3214-3221, 2014.
DOI
[25]
T. Igarashi,; T. Moscovich,; J. F Hughes,. As-rigid-as-possible shape manipulation. ACM Transactions on Graphics Vol. 24, No. 3, 1134-1141, 2005.
DOI Google Scholar
[26]
N. Arad,; D Reisfeld,. Image warping using few anchor points and radial functions. Computer Graphics Forum Vol. 14, No. 1, 35-46, 1995.
DOI Google Scholar
[27]
S. Schaefer,; T. McPhail,; J Warren,. Image deformation using moving least squares. ACM Transactions on Graphics Vol. 25, No. 3, 533-540, 2006.
DOI Google Scholar
[28]
P. Kaufmann,; O. Wang,; A. Sorkine-Hornung,; O. Sorkine-Hornung,; A. Smolic,; M Gross,. Finite element image warping. Computer Graphics Forum Vol. 32, No. 2, 31-39, 2013.
DOI Google Scholar
[29]
C. Rother,; V. Kolmogorov,; A Blake,. “GrabCut”: Interactive foreground extraction using iterated graph cuts. ACM Transactions on Graphics Vol. 23, No. 3, 309-314, 2004.
DOI Google Scholar
[30]
T. Chen,; M.-M. Cheng,; P. Tan,; A. Shamir,; S.-M Hu,. Sketch2Photo: Internet image montage. ACM Transactions on Graphics Vol. 28, No. 5, Article No. 124, 2009.
DOI Google Scholar
[31]
K. He,; J. Sun,; X. Tang, Guided image filtering. In: Computer Vision-ECCV 2010. K. Daniilidis,; P. Maragos,; N. Paragios, Eds. Springer-Verlag Berlin Heidelberg, 1-14, 2010.
DOI
[32]
P. Pérez,; M. Gangnet,; A Blake,. Poisson image editing. ACM Transactions on Graphics Vol. 22, No. 3, 313-318, 2003.
DOI Google Scholar
[33]
E. Shahrian,; D. Rajan, Weighted color and texture sample selection for image matting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 718-725, 2012.
DOI
[34]
S. Belongie,; J. Malik,; J. Puzicha, Shape context: A new descriptor for shape matching and object recognition. In: Proceedings of the 13th International Conference on Neural Information Processing Systems, 798-804, 2000.
[35]
R. Shilkrot,; D Cohen-Or,; A. Shamir,; L Liu,. Garment personalization via identity transfer. IEEE Computer Graphics and Applications Vol. 33, No. 4, 62-72, 2013.
DOI Google Scholar
Publication history
Copyright
Acknowledgements
Rights and permissions

Publication history

Revised: 17 March 2017
Accepted: 09 April 2017
Published: 08 May 2017
Issue date: December 2017

Copyright

© The Author(s) 2017

Acknowledgements

This work was supported by the National Natural Science Foundation of China (Project No. 61521002), and Research Grant of Beijing Higher Institution Engineering Research Center. This work was finished during Zhao-Heng Zheng and Hao-Tian Zhang were undergraduate students in the Department of Computer Science and Technology at Tsinghua University.

Rights and permissions

This article is published with open access at Springerlink.com

The articles published in this journal are distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.

Return