Journal Home > Volume 3 , Issue 3
Computational Visual Media 2017, 3(3): 273-284 https://doi.org/10.1007/s41095-016-0074-0
Research Article | Open Access | Issue | Published: 15 March 2017

Practical automatic background substitution for live video

Show Author's Information Haozhi Huang1,  Xiaonan Fang1,  Yufei Ye1,  Songhai Zhang1( ),  Paul L. Rosin2
Department of Computer Science, Tsinghua University, Beijing, 100084, China.
School of Computer Science and Informatics, Cardiff University, Cardiff, CF24 3AA, UK.
Keywords:
alpha matting, background substitution, background replacement, background subtraction
Cite this article:
Huang H, Fang X, Ye Y, et al. Practical automatic background substitution for live video. Computational Visual Media, 2017, 3(3): 273-284. https://doi.org/10.1007/s41095-016-0074-0
Download citation
EndNote(RIS)
BibTeX

627

Views

14

Downloads

Citations

14

Crossref

N/A

WoS

15

Scopus

2

CSCD

Abstract Full text Electronic supplementary material About this article

In this paper we present a novel automatic background substitution approach for live video. The objective of background substitution is to extract the foreground from the input video and then combine it with a new background. In this paper, we use a color line model to improve the Gaussian mixture model in the background cut method to obtain a binary foreground segmentation result that is less sensitive to brightness differences. Based on the high quality binary segmentation results, we can automatically create a reliable trimap for alpha matting to refine the segmentation boundary. To make the composition result more realistic, an automatic foreground color adjustment step is added to make the foreground look consistent with the new background. Compared to previous approaches, our method can produce higher quality binary segmentation results, and to the best of our knowledge, this is the first time such an automatic and integrated background substitution system has been proposed which can run in real time, which makes it practical for everyday applications.


menu
Abstract
Full text
Outline
Electronic supplementary material
About this article

Practical automatic background substitution for live video

Show Author's information Haozhi Huang1, Xiaonan Fang1, Yufei Ye1, Songhai Zhang1( ), Paul L. Rosin2
Department of Computer Science, Tsinghua University, Beijing, 100084, China.
School of Computer Science and Informatics, Cardiff University, Cardiff, CF24 3AA, UK.

Abstract

In this paper we present a novel automatic background substitution approach for live video. The objective of background substitution is to extract the foreground from the input video and then combine it with a new background. In this paper, we use a color line model to improve the Gaussian mixture model in the background cut method to obtain a binary foreground segmentation result that is less sensitive to brightness differences. Based on the high quality binary segmentation results, we can automatically create a reliable trimap for alpha matting to refine the segmentation boundary. To make the composition result more realistic, an automatic foreground color adjustment step is added to make the foreground look consistent with the new background. Compared to previous approaches, our method can produce higher quality binary segmentation results, and to the best of our knowledge, this is the first time such an automatic and integrated background substitution system has been proposed which can run in real time, which makes it practical for everyday applications.

Keywords: alpha matting, background substitution, background replacement, background subtraction

References(42)

[1]
X. Bai,; J. Wang,; D. Simons,; G. Sapiro, Video SnapCut: Robust video object cutout using localized classifiers. ACM Transactions on Graphics Vol. 28, No. 3, Article No. 70, 2009.
DOI Google Scholar
[2]
T. Chen,; J.-Y. Zhu,; A. Shamir,; S.-M. Hu, Motion-aware gradient domain video composition. IEEE Transactions on Image Processing Vol. 22, No. 7, 2532-2544, 2013.
DOI Google Scholar
[3]
Z. Liu,; M. Cohen, Head-size equalization for better visual perception of video conferencing. In: Proceedings of the IEEE International Conference on Multimedia and Expo, 4, 2005.
[4]
Z. Zhu,; R. R. Martin,; R. Pepperell,; A. Burleigh, 3D modeling and motion parallax for improved videoconferencing. Computational Visual Media Vol. 2, No. 2, 131-142, 2016.
DOI Google Scholar
[5]
D. W. F. Van Krevelen,; R. Poelman, A survey of augmented reality technologies, applications and limitations. International Journal of Virtual Reality Vol. 9, No. 2, 1-21, 2010.
DOI Google Scholar
[6]
N. Apostoloff,; A. Fitzgibbon, Bayesian video matting using learnt image priors. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Vol. 1, I-407-I-414, 2004.
[7]
T. Bouwmans,; F. El Baf,; B. Vachon, Background modeling using mixture of Gaussians for foreground detection—A survey. Recent Patents on Computer Science Vol. 1, No. 3, 219-237, 2008.
DOI Google Scholar
[8]
L. Maddalena,; A. Petrosino, A self-organizing approach to background subtraction for visual surveillance applications. IEEE Transactions on Image Processing Vol. 17, No. 7, 1168-1177, 2008.
DOI Google Scholar
[9]
D.-M. Tsai,; S.-C. Lai, Independent component analysis-based background subtraction for indoor surveillance. IEEE Transactions on Image Processing Vol. 18, No. 1, 158-167, 2009.
DOI Google Scholar
[10]
O. Barnich,; M. Van Droogenbroeck, ViBe: A universal background subtraction algorithm for video sequences. IEEE Transactions on Image Processing Vol. 20, No. 6, 1709-1724, 2011.
DOI Google Scholar
[11]
M. Hofmann,; P. Tiefenbacher,; G. Rigoll, Background segmentation with feedback: The pixel-based adaptive segmenter. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, 38-43, 2012.
DOI
[12]
J. Sun,; W. Zhang,; X. Tang,; H.-Y. Shum, Background cut. In: Computer Vision-ECCV 2006. A. Leonardis,; H. Bischof,; A. Pinz, Eds. Springer Berlin Heidelberg, 628-641, 2006.
DOI
[13]
A. Criminisi,; G. Cross,; A. Blake,; V. Kolmogorov, Bilayer segmentation of live video. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 53-60, 2006.
[14]
P. Yin,; A. Criminisi,; J. Winn,; I. Essa, Bilayer segmentation of webcam videos using tree-based classifiers. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 33, No. 1, 30-42, 2011.
DOI Google Scholar
[15]
Y.-Y. Chuang,; A. Agarwala,; B. Curless,; D. H. Salesin,; R. Szeliski, Video matting of complex scenes. ACM Transactions on Graphics Vol. 21, No. 3, 243-248, 2002.
DOI Google Scholar
[16]
M. Gong,; Y. Qian,; L. Cheng, Integrated foreground segmentation and boundary matting for live videos. IEEE Transactions on Image Processing Vol. 24, No. 4, 1356-1370, 2015.
DOI Google Scholar
[17]
P. Pérez,; M. Gangnet,; A. Blake, Poisson image editing. ACM Transactions on Graphics Vol. 22, No. 3, 313-318, 2003.
DOI Google Scholar
[18]
J. Jia,; J. Sun,; C.-K. Tang,; H.-Y. Shum, Drag-and-drop pasting. ACM Transactions on Graphics Vol. 25, No. 3, 631-637, 2006.
DOI Google Scholar
[19]
G. Buchsbaum, A spatial processor model for object colour perception. Journal of the Franklin Institute Vol. 310, No. 1, 1-26, 1980.
DOI Google Scholar
[20]
G. D. Finlayson,; S. D. Hordley,; P. M. Hubel, Color by correlation: A simple, unifying framework for color constancy. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 23, No. 11, 1209-1221, 2001.
DOI Google Scholar
[21]
D. Cheng,; D. K. Prasad,; M. S. Brown, Illuminant estimation for color constancy: Why spatial-domain methods work and the role of the color distribution. Journal of the Optical Society of America A Vol. 31, No. 5, 1049-1058, 2014.
DOI Google Scholar
[22]
D. Cheng,; B. Price,; S. Cohen,; M. S. Brown, Beyond white: Ground truth colors for color constancy correction. In: Proceedings of the IEEE International Conference on Computer Vision, 298-306, 2015.
DOI
[23]
I. Omer,; M. Werman, Color lines: Image specific color representation. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Vol. 2, II-946-II-953, 2004.
[24]
A. Levin,; D. Lischinski,; Y. Weiss, A closed-form solution to natural image matting. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 30, No. 2, 228-242, 2008.
DOI Google Scholar
[25]
E. H. Land,; J. J. McCann, Lightness and retinex theory. Journal of the Optical Society of America Vol. 61, No. 1, 1-11, 1971.
DOI Google Scholar
[26]
Y. Zhang,; Y.-L. Tang,; K.-L. Cheng, Efficient video cutout by paint selection. Journal of Computer Science and Technology Vol. 30, No. 3, 467-477, 2015.
DOI Google Scholar
[27]
A. R. Smith,; J. F. Blinn, Blue screen matting. In: Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, 259-268, 1996.
DOI
[28]
A. Mumtaz,; W. Zhang,; A. B. Chan, Joint motion segmentation and background estimation in dynamic scenes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 368-375, 2014.
DOI
[29]
L. Zhang,; H. Huang,; H. Fu, EXCOL: An extract-and-complete layering approach to cartoon animation reusing. IEEE Transactions on Visualization and Computer Graphics Vol. 18, No. 7, 1156-1169, 2012.
DOI Google Scholar
[30]
E. S. L. Gastal,; M. M. Oliveira, Shared sampling for real-time alpha matting. Computer Graphics Forum Vol. 29, No. 2, 575-584, 2010.
DOI Google Scholar
[31]
X. Chen,; D. Zou,; S. Zhou,; Q. Zhao,; P. Tan, Image matting with local and nonlocal smooth priors. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1902-1907, 2013.
DOI
[32]
B.-Y. Wong,; K.-T. Shih,; C.-K. Liang,; H. H. Chen, Single image realism assessment and recoloring by color compatibility. IEEE Transactions on Multimedia Vol. 14, No. 3, 760-769, 2012.
DOI Google Scholar
[33]
D. Cohen-Or,; O. Sorkine,; R. Gal,; T. Leyvand,; Y.-Q. Xu, Color harmonization. ACM Transactions on Graphics Vol. 25, No. 3, 624-630, 2006.
DOI Google Scholar
[34]
Z. Kuang,; P. Lu,; X. Wang,; X. Lu, Learning self-adaptive color harmony model for aesthetic quality classification. In: Proceedings of SPIE 9443, the 6th International Conference on Graphic and Image Processing, 94431O, 2015.
DOI
[35]
T. Chen,; M.-M. Cheng,; P. Tan,; A. Shamir,; S.-M. Hu, Sketch2Photo: Internet image montage. ACM Transactions on Graphics Vol. 28, No. 5, Article No. 124, 2009.
DOI Google Scholar
[36]
Z. Farbman,; G. Hoffer,; Y. Lipman,; D. Cohen-Or,; D. Lischinski, Coordinates for instant image cloning. ACM Transactions on Graphics Vol. 28, No. 3, Article No. 67, 2009.
DOI Google Scholar
[37]
Y. Boykov,; V. Kolmogorov, An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. In: Energy Minimization Methods in Computer Vision and Pattern Recognition. M. Figueiredo,; J. Zerubia,; A. K. Jain, Eds. Springer Berlin Heidelberg, 359-374, 2001.
DOI
[38]
M. H. Sigari,; N. Mozayani,; H. R. Pourreza, Fuzzy running average and fuzzy background subtraction: concepts and application. International Journal of Computer Science and Network Security Vol. 8, No. 2, 138-143, 2008.
Google Scholar
[39]
A. Sobral, BGSLibrary. 2016. Available at https://github.com/andrewssobral/bgslibrary.
[40]
P. L. Rosin,; E. Ioannidis, Evaluation of global image thresholding for change detection. Pattern Recognition Letters Vol. 24, No. 14, 2345-2356, 2003.
DOI Google Scholar
[41]
D. J. Kerbyson,; T. J. Atherton, Circle detection using Hough transform filters. In: Proceedings of the 5th International Conference on Image Processing and its Applications, 370-374, 1995.
DOI
[42]
Graphics and Media Lab. Videomatting benchmark. 2016. Available at http://videomatting.com.
Video
41095_2016_74_MOESM1_ESM.mp4
Publication history
Copyright
Acknowledgements
Rights and permissions

Publication history

Revised: 18 August 2016
Accepted: 20 December 2016
Published: 15 March 2017
Issue date: September 2017

Copyright

© The Author(s) 2016

Acknowledgements

We thank the reviewers for their valuable comments. This work was supported by the National High-Tech R&D Program of China (Project No. 2012AA011903), the National Natural Science Foundation of China (Project No. 61373069), the Research Grant of Beijing Higher Institution Engineering Research Center, and Tsinghua-Tencent Joint Laboratory for Internet Innovation Technology.

Rights and permissions

This article is published with open access at Springerlink.com

The articles published in this journal are distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.

Return