Journal Home > Volume 9 , Issue 1
Computational Visual Media 2023, 9(1): 141-154 https://doi.org/10.1007/s41095-022-0273-9
Research Article | Open Access | Issue | Published: 18 October 2022

Joint specular highlight detection and removal in single images via Unet-Transformer

Show Author's Information Zhongqi Wu1,2,  Jianwei Guo1,2( ),  Chuanqing Zhuang2,  Jun Xiao2( ),  Dong-Ming Yan1,2,  Xiaopeng Zhang1,2
National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
The School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
Keywords:
specular highlight detection, specular high-light removal, Unet-Transformer
Cite this article:
Wu Z, Guo J, Zhuang C, et al. Joint specular highlight detection and removal in single images via Unet-Transformer. Computational Visual Media, 2023, 9(1): 141-154. https://doi.org/10.1007/s41095-022-0273-9
Download citation
EndNote(RIS)
BibTeX

879

Views

59

Downloads

Citations

11

Crossref

8

WoS

11

Scopus

0

CSCD

Abstract Full text About this article

Specular highlight detection and removal is a fundamental problem in computer vision and image processing. In this paper, we present an efficient end-to-end deep learning model for automatically detecting and removing specular highlights in a single image. In particular, an encoder–decoder network is utilized to detect specular highlights, and then a novel Unet-Transformer network performs highlight removal; we append transformer modules instead of feature maps in the Unet architecture. We also introduce a highlight detection module as a mask to guide the removal task. Thus, these two networks can be jointly trained in an effective manner. Thanks to the hierarchical and global properties of the transformer mechanism, our framework is able to establish relationships between continuous self-attention layers, making it possible to directly model the mapping between the diffuse area and the specular highlight area, and reduce indeterminacy within areas containing strong specular highlight reflection. Experiments on public benchmark and real-world images demonstrate that our approach outperforms state-of-the-art methods for both highlight detection and removal tasks.


menu
Abstract
Full text
Outline
About this article

Joint specular highlight detection and removal in single images via Unet-Transformer

Show Author's information Zhongqi Wu1,2, Jianwei Guo1,2( ), Chuanqing Zhuang2, Jun Xiao2( ), Dong-Ming Yan1,2, Xiaopeng Zhang1,2
National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
The School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China

Abstract

Specular highlight detection and removal is a fundamental problem in computer vision and image processing. In this paper, we present an efficient end-to-end deep learning model for automatically detecting and removing specular highlights in a single image. In particular, an encoder–decoder network is utilized to detect specular highlights, and then a novel Unet-Transformer network performs highlight removal; we append transformer modules instead of feature maps in the Unet architecture. We also introduce a highlight detection module as a mask to guide the removal task. Thus, these two networks can be jointly trained in an effective manner. Thanks to the hierarchical and global properties of the transformer mechanism, our framework is able to establish relationships between continuous self-attention layers, making it possible to directly model the mapping between the diffuse area and the specular highlight area, and reduce indeterminacy within areas containing strong specular highlight reflection. Experiments on public benchmark and real-world images demonstrate that our approach outperforms state-of-the-art methods for both highlight detection and removal tasks.

Keywords: specular highlight detection, specular high-light removal, Unet-Transformer

References(60)

[1]
Arbeláez, P.; Maire, M.; Fowlkes, C.; Malik, J. Contour detection and hierarchical image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 33, No. 5, 898–916, 2011.
DOI Google Scholar
[2]
Tao, M. W.; Su, J. C.; Wang, T. C.; Malik, J.; Ramamoorthi, R. Depth estimation and specularremoval for glossy surfaces using point and line consistency with light-field cameras. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 38, No. 6, 1155–1169, 2016.
DOI Google Scholar
[3]
Ramadan, H.; Lachqar, C.; Tairi, H. A survey of recent interactive image segmentation methods. Computational Visual Media Vol. 6, No. 4, 355–384, 2020.
DOI Google Scholar
[4]
Khanian, M.; Boroujerdi, A. S.; Breuß, M. Photometric stereo for strong specular highlights. Computational Visual Media Vol. 4, No. 1, 83–102, 2018.
DOI Google Scholar
[5]
Cui, Z. P.; Gu, J. W.; Shi, B. X.; Tan, P.; Kautz, J. Polarimetric multi-view stereo. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 369–378, 2017.
DOI
[6]
Xue, M. L.; Shivakumara, P.; Zhang, C.; Xiao, Y.; Lu, T.; Pal, U.; Lopresti, D.; Yang, Z. Arbitrarily-oriented text detection in low light natural scene images. IEEE Transactions on Multimedia Vol. 23, 2706–2720, 2021.
DOI Google Scholar
[7]
Osadchy, M.; Jacobs, D. W.; Ramamoorthi, R. Using specularities for recognition. In: Proceedings of the 9th IEEE International Conference on Computer Vision, 1512–1519, 2003.
DOI
[8]
Li, R. Y.; Pan, J. J.; Si, Y. Q.; Yan, B.; Hu, Y.; Qin, H. Specular reflections removal for endoscopic image sequences with adaptive-RPCA decomposition. IEEE Transactions on Medical Imaging Vol. 39, No. 2, 328–340, 2020.
DOI Google Scholar
[9]
Artusi, A.; Banterle, F.; Chetverikov, D. A survey of specularity removal methods. Computer Graphics Forum Vol. 30, No. 8, 2208–2230, 2011.
DOI Google Scholar
[10]
Shafer, S. A. Using color to separate reflection components. Color Research & Application Vol. 10, No. 4, 210–218, 1985.
DOI Google Scholar
[11]
Shen, H. L.; Zhang, H. G.; Shao, S. J.; Xin, J. H. Chromaticity-based separation of reflection components in a single image. Pattern Recognition Vol. 41, No. 8, 2461–2469, 2008.
DOI Google Scholar
[12]
Brainard, D. H.; Freeman, W. T. Bayesian color constancy. Journal of the Optical Society of America A Vol. 14, No. 7, 1393–1411, 1997.
DOI Google Scholar
[13]
Finlayson, G. D.; Hordley, S. D.; HubeL, P. M. Color by correlation: A simple, unifying framework for color constancy. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 23, No. 11, 1209–1221, 2001.
DOI Google Scholar
[14]
Tan, R. T.; Nishino, K.; Ikeuchi, K. Color constancy through inverse-intensity chromaticity space. Journal of the Optical Society of America A Vol. 21, No. 3, 321–334, 2004.
DOI Google Scholar
[15]
Shi, J.; Dong, Y.; Su, H.; Yu, S. X. Learning non-Lambertian object intrinsics across ShapeNet categories. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 5844–5853, 2017.
DOI
[16]
Lin, J.; El Amine Seddik, M.; Tamaazousti, M.; Tamaazousti, Y.; Bartoli, A. Deep multi-class adversarial specularity removal. In: Image Analysis. Lecture Notes in Computer Science, Vol. 11482. Felsberg, M.; Forssén, P. E.; Sintorn, I. M.; Unger, J. Eds. Springer Cham, 3–15, 2019.
[17]
Muhammad, S.; Dailey, M. N.; Farooq, M.; Majeed, M. F.; Ekpanyapong, M. Spec-Net and Spec-CGAN: Deep learning models for specularity removal from faces. Image and Vision Computing Vol. 93, 103823, 2020.
DOI Google Scholar
[18]
Xu, Y. F.; Wei, H. P.; Lin, M. X.; Deng, Y. Y.; Sheng, K. K.; Zhang, M. D.; Tang, F.; Dong, W.; Huang, F.; Xu, C. Transformers in computational visual media: A survey. Computational Visual Media Vol. 8, No. 1, 33–62, 2022.
DOI Google Scholar
[19]
Liu, Z.; Lin, Y. T.; Cao, Y.; Hu, H.; Wei, Y. X.; Zhang, Z.; Lin, S.; Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 9992–10002, 2021.
DOI
[20]
Maloney, L. T.; Wandell, B. A. Color constancy: A method for recovering surface spectral reflectance. Journal of the Optical Society of America A Vol. 3, No. 1, 29–33, 1986.
DOI Google Scholar
[21]
Park, J. B.; Kak, A. C. A truncated least squares approach to the detection of specular highlights in color images. In: Proceedings of the IEEE International Conference on Robotics and Automation, 1397–1403, 2003.
[22]
Meslouhi, O.; Kardouchi, M.; Allali, H.; Gadi, T.; Benkaddour, Y. Automatic detection and inpainting of specular reflections for colposcopic images. Central European Journal of Computer Science Vol. 1, No. 3, 341–354, 2011.
DOI Google Scholar
[23]
Zhang, W. M.; Zhao, X.; Morvan, J. M.; Chen, L. M. Improving shadow suppression for illumination robust face recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 41, No. 3, 611–624, 2019.
DOI Google Scholar
[24]
Hoyer, P. O. Non-negative matrix factorization with sparseness constraints. Journal of Machine Learning Research Vol. 5, 1457–1469, 2004.
Google Scholar
[25]
Fu, G.; Zhang, Q.; Lin, Q. F.; Zhu, L.; Xiao, C. X. Learning to detect specular highlights from real-world images. In: Proceedings of the 28th ACM International Conference on Multimedia, 1873–1881, 2020.
DOI
[26]
Tan, P.; Quan, L.; Lin, S. Separation of highlight reflections on textured surfaces. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 1855–1860, 2006.
[27]
Shen, H.-L.; Cai, Q.-Y. Simple and efficient method for specularity removal in an image. Applied Optics Vol. 48, No. 14, 2711, 2009.
DOI Google Scholar
[28]
Shen, H. L.; Zheng, Z. H. Real-time highlight removal using intensity ratio. Applied Optics Vol. 52, No. 19, 4483–4493, 2013.
DOI Google Scholar
[29]
Yang, J. W.; Liu, L. X.; Li, S. Z. Separating specular and diffuse reflection components in the HSI color space. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, 891–898, 2013.
DOI
[30]
Yang, Q. X.; Tang, J. H.; Ahuja, N. Efficient and robust specular highlight removal. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 37, No. 6, 1304–1311, 2015.
DOI Google Scholar
[31]
Akashi, Y.; Okatani, T. Separation of reflection components by sparse non-negative matrix factorization. In: Computer Vision – ACCV 2014. Lecture Notes in Computer Science, Vol. 9007. Cremers, D.; Reid, I.; Saito, H.; Yang, M. H. Eds. Springer Cham, 611–625, 2015.
[32]
Guo, J.; Zhou, Z. J.; Wang, L. M. Single image highlight removal with a sparse and low-rank reflection model. In: Computer Vision – ECCV 2018. Lecture Notes in Computer Science, Vol. 11208. Ferrari, V.; Hebert, M.; Sminchisescu, C.; Weiss, Y. Eds. Springer Cham, 282–298, 2018.
[33]
Fu, G.; Zhang, Q.; Song, C. F.; Lin, Q. F.; Xiao, C. X. Specular highlight removal for real-world images. Computer Graphics Forum Vol. 38, No. 7, 253–263, 2019.
DOI Google Scholar
[34]
Nayar, S. K.; Fang, X. S.; Boult, T. Separation of reflection components using color and polarization. International Journal of Computer Vision Vol. 21, No. 163–186, 1997.
Google Scholar
[35]
Umeyama, S.; Godin, G. Separation of diffuse and specular components of surface reflection by use of polarization and statistical analysis of images. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 26, No. 5, 639–647, 2004.
DOI Google Scholar
[36]
Wang, F.; Ainouz, S.; Petitjean, C.; Bensrhair, A. Specularity removal: A global energy minimization approach based on polarization imaging. Computer Vision and Image Understanding Vol. 158, 31–39, 2017.
DOI Google Scholar
[37]
Wen, S.; Zheng, Y.; Lu, F. Polarization guided specular reflection separation. IEEE Transactions on Image Processing Vol. 30, 7280–7291, 2021.
DOI Google Scholar
[38]
Sapiro, G. Color and illuminant voting. IEEE Trans-actions on Pattern Analysis and Machine Intelligence Vol. 21, No. 11, 1210–1215, 1999.
DOI Google Scholar
[39]
Imai, Y.; Kato, Y.; Kadoi, H.; Horiuchi, T.; Tominaga, S. Estimation of multiple illuminants based on specular highlight detection. In: Computational Color Imaging. Lecture Notes in Computer Science, Vol. 6626. Schettini, R.; Tominaga, S.; Trémeau, A. Eds. Springer Berlin Heidelberg, 85–98, 2011.
[40]
Forsyth, D. A. A novel algorithm for color constancy. International Journal of Computer Vision Vol. 5, No. 1, 5–35, 1990.
DOI Google Scholar
[41]
Hansen, T.; Olkkonen, M.; Walter, S.; Gegenfurtner, K. R. Memory modulates color appearance. Nature Neuroscience Vol. 9, No. 11, 1367–1368, 2006.
DOI Google Scholar
[42]
Joze, H. R. V.; Drew, M. S. Exemplar-based color constancy and multiple illumination. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 36, No. 5, 860–873, 2014.
DOI Google Scholar
[43]
Lin, P.; Quan, L.; Shum, H.-Y. Highlight removal by illumination-constrained inpainting. In: Proceedings of the 9th IEEE International Conference on Computer Vision, 164–169, 2003.
[44]
Tan, R. T.; Ikeuchi, K. Separating reflection components of textured surfaces using a single image. In: Digitally Archiving Cultural Objects. Springer Boston MA, 353–384, 2008.
DOI
[45]
Tan, T. T.; Nishino, K.; Ikeuchi, K. Illumination chromaticity estimation using inverse-intensity chromaticity space. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, I, 2003.
DOI
[46]
Funke, I.; Bodenstedt, S.; Riediger, C.; Weitz, J.; Speidel, S. Generative adversarial networks for specular highlight removal in endoscopic images. In: Proceedings of the SPIE 10576, Medical Imaging 2018: Image-Guided Procedures, Robotic Interventions, and Modeling, 1057604, 2018.
DOI
[47]
Wu, Z. Q.; Zhuang, C. Q.; Shi, J.; Xiao, J.; Guo, J. W. Deep specular highlight removal for single real-world image. In: Proceedings of the SIGGRAPH Asia 2020 Posters, Article No. 34, 2020.
[48]
Wu, Z. Q.; Zhuang, C. Q.; Shi, J.; Guo, J. W.; Xiao, J.; Zhang, X. P.; Yan, D.-M. Single-image specular highlight removal via real-world dataset construction. IEEE Transactions on Multimedia Vol. 24, 3782–3793, 2022.
DOI Google Scholar
[49]
Yi, R. J.; Tan, P.; Lin, S. Leveraging multi-view image sets for unsupervised intrinsic image decomposition and highlight separation. Proceedings of the AAAI Conference on Artificial Intelligence Vol. 34, No. 7, 12685–12692, 2020.
DOI Google Scholar
[50]
Fu, G.; Zhang, Q.; Zhu, L.; Li, P.; Xiao, C. X. A multi-task network for joint specular highlight detection and removal. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 7748–7757, 2021.
DOI
[51]
Ronneberger, O.; Fischer, P.; Brox, T. U-Net: Convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015. Lecture Notes in Computer Science, Vol. 9351. Navab, N.; Hornegger, J.; Wells, W.; Frangi, A. Eds. Springer Cham, 234–241, 2015.
DOI
[52]
Cao, H.; Wang, Y.; Chen, J.; Jiang, D.; Zhang, X.; Tian, Q.; Wang, M. Swin-Unet: Unet-like pure transformer for medical image segmentation. arXiv preprint arXiv: 2105.05537, 2021.
Google Scholar
[53]
Wei, K. X.; Yang, J. L.; Fu, Y.; Wipf, D.; Huang, H. Single image reflection removal exploiting misaligned training data and network enhancements. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8170–8179, 2019.
DOI
[54]
Gatys, L. A.; Ecker, A. S.; Bethge, M. Image style transfer using convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2414–2423, 2016.
DOI
[55]
Simonyan, K.; Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409.1556, 2014.
Google Scholar
[56]
Murmann, L.; Gharbi, M.; Aittala, M.; Durand, F. A dataset of multi-illumination images in the wild. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 4079–4088, 2019.
DOI
[57]
Guo, X. J.; Cao, X. C.; Ma, Y. Robust separation of reflection from multiple images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2195–2202, 2014.
DOI
[58]
Kingma, D. P.; Ba, J. Adam: A method for stochastic optimization. In: Proceedings of the 3rd International Conference for Learning Representations, 2015.
[59]
Yamamoto, T.; Kitajima, T.; Kawauchi, R. Efficient improvement method for separation of reflection components based on an energy function. In: Proceedings of the IEEE International Conference on Image Processing, 4222–4226, 2017.
DOI
[60]
Hou, S.; Wang, C.; Quan, W.; Jiang, J.; Yan, D. M. Text-aware single image specular highlight removal. In: Pattern Recognition and Computer Vision. Lecture Notes in Computer Science, Vol. 13022. Springer Cham, 115–127, 2021.
Publication history
Copyright
Acknowledgements
Rights and permissions

Publication history

Received: 10 November 2021
Accepted: 03 February 2022
Published: 18 October 2022
Issue date: March 2023

Copyright

© The Author(s) 2022.

Acknowledgements

This work was partially funded by the National Natural Science Foundation of China (U21A20515, 62172416, 62172415, U2003109), and Youth Innovation Promotion Association of the Chinese Academy of Sciences (2022131).

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduc-tion in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.

Return