Journal Home > Volume 5 , Issue 1
Computational Visual Media 2019, 5(1): 105-115 https://doi.org/10.1007/s41095-019-0136-1
Research Article | Open Access | Issue | Published: 08 April 2019

ShadowGAN: Shadow synthesis for virtual objects with conditional adversarial networks

Show Author's Information Shuyang Zhang1( ),  Runze Liang2,  Miao Wang3
University of Michigan, Ann Arbor, MI 48109 USA.
Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China.
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing 100191, China.
Keywords:
deep learning, shadow synthesis, genera-tive adversarial networks, image synthesis
Cite this article:
Zhang S, Liang R, Wang M. ShadowGAN: Shadow synthesis for virtual objects with conditional adversarial networks. Computational Visual Media, 2019, 5(1): 105-115. https://doi.org/10.1007/s41095-019-0136-1
Download citation
EndNote(RIS)
BibTeX

516

Views

36

Downloads

Citations

38

Crossref

N/A

WoS

44

Scopus

0

CSCD

Abstract Full text About this article

We introduce ShadowGAN, a generative adversarial network (GAN) for synthesizing shadows for virtual objects inserted in images. Given a target image containing several existing objects with shadows, and an input source object with a specified insertionposition, the network generates a realistic shadow for the source object. The shadow is synthesized by a generator; using the proposed local adversarial and global adversarial discriminators, the synthetic shadow’s appearance is locally realistic in shape, and globally consistent with other objects’ shadows in terms of shadow direction and area. To overcome the lack of training data, we produced training samples based on public 3D models and rendering technology. Experimental results from a user study show that the synthetic shadowed results look natural and authentic.


menu
Abstract
Full text
Outline
About this article

ShadowGAN: Shadow synthesis for virtual objects with conditional adversarial networks

Show Author's information Shuyang Zhang1( ), Runze Liang2, Miao Wang3
University of Michigan, Ann Arbor, MI 48109 USA.
Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China.
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing 100191, China.

Abstract

We introduce ShadowGAN, a generative adversarial network (GAN) for synthesizing shadows for virtual objects inserted in images. Given a target image containing several existing objects with shadows, and an input source object with a specified insertionposition, the network generates a realistic shadow for the source object. The shadow is synthesized by a generator; using the proposed local adversarial and global adversarial discriminators, the synthetic shadow’s appearance is locally realistic in shape, and globally consistent with other objects’ shadows in terms of shadow direction and area. To overcome the lack of training data, we produced training samples based on public 3D models and rendering technology. Experimental results from a user study show that the synthetic shadowed results look natural and authentic.

Keywords: deep learning, shadow synthesis, genera-tive adversarial networks, image synthesis

References(38)

[1]
K. Karsch,; V. Hedau,; D. Forsyth,; D. Hoiem, Rendering synthetic objects into legacy photographs. ACM Transactions on Graphics Vol. 30, No. 6, Article No. 157, 2011.
DOI Google Scholar
[2]
K. Karsch,; K. Sunkavalli,; S. Hadap,; N. Carr,; H. Jin,; R. Fonte,; M. Sittig,; D. Forsyth, Automatic scene inference for 3D object compositing. ACM Transactions on Graphics Vol. 33, No. 3, Article No. 32, 2014.
DOI Google Scholar
[3]
E. Kee,; J. F. O’Brien,; H. Farid, Exposing photo manipulation from shading and shadows. ACM Transactions on Graphics Vol. 33, No. 5, Article No. 165, 2014.
DOI Google Scholar
[4]
B. Liu,; K. Xu,; R. R. Martin, Static scene illumination estimation from videos with applications. Journal of Computer Science and Technology Vol. 32, No. 3, 430-442, 2017.
DOI Google Scholar
[5]
S. Bell,; K. Bala,; N. Snavely, Intrinsic images in the wild. ACM Transactions on Graphics Vol. 33, No. 4, Article No. 159, 2014.
DOI Google Scholar
[6]
S. Bi,; X. Han,; Y. Yu, An L1 image transform for edge-preserving smoothing and scene-level intrinsic decomposition. ACM Transactions on Graphics Vol. 34, No. 4, Article No. 78, 2015.
DOI Google Scholar
[7]
A. Bousseau,; S. Paris,; F. Durand, User-assisted intrinsic images. ACM Transactions on Graphics Vol. 28, No. 5, Article No. 130, 2009.
DOI Google Scholar
[8]
Q. Fan,; J. Yang,; G. Hua,; B. Chen,; D. Wipf, Revisiting deep intrinsic image decompositions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 8944-8952, 2018.
DOI
[9]
A. Panagopoulos,; D. Samaras,; N. Paragios, Robust shadow and illumination estimation using a mixture model. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 651-658, 2009.
DOI
[10]
I. Sato,; Y. Sato,; K. Ikeuchi, Illumination from shadows. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 25, No. 3, 290-300, 2003.
DOI Google Scholar
[11]
E. A. Khan,; E. Reinhard,; R. W. Fleming,; H. H. Bulthoff, Image-based material editing. ACM Transactions on Graphics Vol. 25, No. 3, 654-663, 2006.
DOI Google Scholar
[12]
S. Ge,; X. Jin,; Q. Ye,; Z. Luo,; Q. Li, Image editing by object-aware optimal boundary searching and mixed-domain composition. Computational Visual Media Vol. 4, No. 1, 71-82, 2018.
DOI Google Scholar
[13]
J. Kronander,; F. Banterle,; A. Gardner,; E. Miandji,; J. Unger, Photorealistic rendering of mixed reality scenes. Computer Graphics Forum Vol. 34, 643-665, 2015.
DOI Google Scholar
[14]
P. Debevec, Rendering synthetic objects into real scenes: Bridging traditional and image-based graphics with global illumination and high dynamic range photography. In: Proceedings of the ACM SIGGRAPH 2008 Classes, Article No. 32, 2008.
DOI
[15]
Y. Shor,; D. Lischinski, The shadow meets the mask: Pyramid-based shadow removal. Computer Graphics Forum Vol. 27, No. 2, 577-586, 2008.
DOI Google Scholar
[16]
J. Wang,; X. Li,; J. Yang, Stacked conditional generative adversarial networks for jointly learning shadow detection and shadow removal. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1788-1797, 2018.
DOI
[17]
L. Qu,; J. Tian,; S. He,; Y. Tang,; R. W. H. Lau. DeshadowNet: A multi-context embedding deep network for shadow removal. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4067-4075, 2017.
DOI
[18]
L. Xu,; F. Qi,; R. Jiang, Shadow removal from a single image. In: Proceedings of the 6th International Conference on Intelligent Systems Design andApplications, 1049-1054, 2006.
DOI
[19]
R. Guo,; Q. Dai,; D. Hoiem, Single-image shadow detection and removal using paired regions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2033-2040, 2011.
DOI
[20]
L. Zhang,; Q. Zhang,; C. Xiao, Shadow remover: Image shadow removal based on illumination recovering optimization. IEEE Transactions on Image Processing Vol. 24, No. 11, 4623-4636, 2015.
DOI Google Scholar
[21]
L.-Q. Ma,; J. Wang,; E. Shechtman,; K. Sunkavalli,; S.-M. Hu, Appearance harmonization for single image shadow removal. Computer Graphics Forum Vol. 35, No. 7, 189-197, 2016.
DOI Google Scholar
[22]
I. Goodfellow,; J. Pouget-Abadie,; M. Mirza,; B. Xu,; D. Warde-Farley,; S. Ozair,; A. Courville,; Y. Bengio, Generative adversarial nets. In: Proceedings of the Advances in Neural Information Processing Systems 27, 2672-2680, 2014.
[23]
M. Li,; W. Zuo,; D. Zhang, Deep identity-aware transfer of facial attributes. arXiv preprint arXiv:1610.05586, 2016.
[24]
Y. Choi,; M. Choi,; M. Kim,; J.-W. Ha,; S. Kim,; J. Choo, StarGAN: Unified generative adversarial networks for multi-domain image-to-image translation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 8789-8797, 2018.
DOI
[25]
W. Shen,; R. Liu, Learning residual images for face attribute manipulation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4030-4038, 2017.
DOI
[26]
C. Ledig,; L. Theis,; F. Huszar,; J. Caballero,; A. Cunningham,; A. Acosta,; A. Aitken,; A. Tejani,; J. Totz,; Z. Wang,; W. Shi, Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4681-4690, 2017.
DOI
[27]
D. Pathak,; P. Krahenbuhl,; J. Donahue,; T. Darrell,; A. A. Efros, Context encoders: Feature learning by inpainting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2536-2544, 2016.
DOI
[28]
S. Iizuka,; E. Simo-Serra,; H. Ishikawa, Globally and locally consistent image completion. ACM Transactions on Graphics Vol. 36, No. 4, Article No. 107, 2017.
DOI Google Scholar
[29]
M. Mirza,; S. Osindero, Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784, 2014.
[30]
P. Isola,; J.-Y. Zhu,; T. Zhou,; A. A. Efros, Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1125-1134, 2017.
DOI
[31]
J.-Y. Zhu,; T. Park,; P. Isola,; A. A. Efros, Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEEInternational Conference on Computer Vision, 2223-2232, 2017.
DOI
[32]
T. Portenier,; Q. Hu,; A. Szabo,; S. A. Bigdeli,; P. Favaro,; M. Zwicker, Faceshop: Deep sketch-based face image editing. ACM Transactions on Graphics Vol. 37, No. 4, Article No. 99, 2018.
DOI Google Scholar
[33]
Z. Wei,; Y. Sun,; J. Lin,; S. Liu, Learning adaptive receptive fields for deep image parsing networks. Computational Visual Media Vol. 4, No. 3, 231-244, 2018.
DOI Google Scholar
[34]
A. X. Chang,; T. Funkhouser,; L. Guibas,; P. Hanrahan,; Q. Huang,; Z. Li,; S. Savarese,; M. Savva,; S. Song,; H. Su,; J. Xiao,; L. Yi,; F. Yu, ShapeNet: An information-rich 3D model repository. arXiv preprint arXiv:1512.03012, 2015.
[35]
E. P. Lafortune,; Y. D. Willems, Bi-directional path tracing. In: Proceedings of the 3rd International Conference on Computational Graphics and Visualiza-tion Techniques, 145-153, 1993.
[36]
A. Shrivastava,; T. Pfister,; O. Tuzel,; J. Susskind,; W. Wang,; R. Webb, Learning from simulated and unsupervised images through adversarial training. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2107-2116, 2017.
DOI
[37]
F. Yu,; V. Koltun, Multi-scale context aggregation by dilated convolutions. arXiv preprint arXiv:1511.07122, 2015.
[38]
D. P. Kingma,; J. Ba, Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Publication history
Copyright
Acknowledgements
Rights and permissions

Publication history

Revised: 28 December 2018
Accepted: 05 February 2019
Published: 08 April 2019
Issue date: March 2019

Copyright

© The Author(s) 2019

Acknowledgements

The authors would like to thank all the reviewers. This work was supported by the National NaturalScience Foundation of China (Project Nos.61561146393 and 61521002), the China Postdoctoral Science Foundation (Project No. 2016M601032), and a Research Grant of Beijing Higher Institution Engineering Research Center.

Rights and permissions

This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduc-tion in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from thecopyright holder.

To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.

Return