AI Chat Paper
Note: Please note that the following content is generated by AMiner AI. SciOpen does not take any responsibility related to this content.
{{lang === 'zh_CN' ? '文章概述' : 'Summary'}}
{{lang === 'en_US' ? '中' : 'Eng'}}
Chat more with AI
Powered by AMiner. Clicking this button will take you away from SciOpen.
Home Computational Visual Media Article
PDF (24.8 MB)
Cite
EndNote(RIS) BibTeX
Collect
Submit Manuscript AI Chat Paper
Show Outline
Outline
Show full outline
Hide outline
Outline
Show full outline
Hide outline
Research Article | Open Access

Lossless intrinsic image decomposition via learning shading feature filtering

Beijing Engineering Research Center of Mixed Reality and Advanced Display, School of Optics and Photonics, Beijing Institute of Technology, Beijing 100081, China
Zhengzhou Research Institute, Beijing Institute of Technology, Zhengzhou 450000, China
Show Author Information

Graphical Abstract

Abstract

Intrinsic image decomposition decomposes an image into reflectance and shading. It has been applied in image editing, augmented reality, and geometry estimation. However, the complete decoupling between reflectance and shading, as well as the consistency of the reconstructed image with the original image, have become the main challenges in the application of intrinsic image decomposition. To improve the performance of the intrinsic image decomposition algorithm for these two challenges, we propose a novel deep learning framework that works separately to learn features unique to different intrinsic images. Based on this framework, we developed more effective loss functions to strengthen the decoupling of reflectance and shading and to maintain the decomposition without losing as much information of the original image as possible. We trained the network on a mixture of synthetic and real datasets and evaluated the results of the experiments on real datasets. The results show that our proposed method not only outperformed existing state-of-the-art methods in qualitative and quantitative comparisons in terms of reflectance but was also competitive in terms of reconstructed consistency and shading. Finally, we implemented several realistic image-editing applications, and the results were visually superior to other results.

Keywords

intrinsic image decompositioncomputer graphicsimage editingconvolutional neural network

References

[1]
Barrow, H. G.; Tenenbaum, J. M. Recovering intrinsic scene characteristics from images. In: Computer Vision Systems. Hanson, A.; Riseman, E. Eds. New York: Academic Press, 1978.
[2]

Garces, E.; Rodriguez-Pardo, C.; Casas, D.; Lopez-Moreno, J. A survey on intrinsic images: Delving deep into lambert and beyond. International Journal of Computer Vision Vol. 130, No. 3, 836–868, 2022.
Crossref Google Scholar
[3]

Guarnera, D.; Guarnera, G. C.; Ghosh, A.; Denk, C.; Glencross, M. BRDF representation and acquisition. Computer Graphics Forum Vol. 35, No. 2, 625–650, 2016.
Crossref Google Scholar
[4]

Bonneel, N.; Kovacs, B.; Paris, S.; Bala, K. Intrinsic decompositions for image editing. Computer Graphics Forum Vol. 36, No. 2, 593–609, 2017.
Crossref Google Scholar
[5]

Luo, J.; Huang, Z.; Li, Y.; Zhou, X.; Zhang, G.; Bao, H. NⅡD-net: Adapting surface normal knowledge for intrinsic image decomposition in indoor scenes. IEEE Transactions on Visualization and Computer Graphics Vol. 26, No. 12, 3434–3445, 2020.
Crossref Google Scholar
[6]

Sheng, B.; Li, P.; Jin, Y.; Tan, P.; Lee, T. Y. Intrinsic image decomposition with step and drift shading separation. IEEE Transactions on Visualization and Computer Graphics Vol. 26, No. 2, 1332–1346, 2020.
Crossref Google Scholar
[7]
Kong, N.; Black, M. J. Intrinsic depth: Improving depth transfer with intrinsic images. In: Proceedings of the IEEE International Conference on Computer Vision, 3514–3522, 2015.
Crossref
[8]

Barron, J. T.; Malik, J. Shape, illumination, and reflectance from shading. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 37, No. 8, 1670–1687, 2015.
Crossref Google Scholar
[9]

Maurer, D.; Ju, Y. C.; Breuß, M.; Bruhn, A. Combining shape from shading and stereo: A joint variational method for estimating depth, illumination and albedo. International Journal of Computer Vision Vol. 126, No. 12, 1342–1366, 2018.
Crossref Google Scholar
[10]

Meka, A.; Zollhöfer, M.; Richardt, C.; Theobalt, C. Live intrinsic video. ACM Transactions on Graphics Vol. 35, No. 4, Article No. 109, 2016.
Crossref Google Scholar
[11]

Meka, A.; Fox, G.; Zollhöfer, M.; Richardt, C.; Theobalt, C. Live user-guided intrinsic video for static scenes. IEEE Transactions on Visualization and Computer Graphics Vol. 23, No. 11, 2447–2454, 2017.
Crossref Google Scholar
[12]
Baslamisli, A. S.; Groenestege, T. T.; Das, P.; Le, H. A.; Karaoglu, S.; Gevers, T. Joint learning of intrinsic images and semantic segmentation. In: Computer Vision – ECCV 2018. Lecture Notes in Computer Science, Vol. 11210. Ferrari, V.; Hebert, M.; Sminchisescu, C.; Weiss, Y. Eds. Springer Cham, 286–302, 2018.
Crossref
[13]

Bi, S.; Han, X.; Yu, Y. An L1 image transform for edge-preserving smoothing and scene-level intrinsic decomposition. ACM Transactions on Graphics Vol. 34, No. 4, Article No. 78, 2015.
Crossref Google Scholar
[14]
Bell, M.; Freeman, E. T. Learning local evidence for shading and reflectance. In: Proceedings of the 8th IEEE International Conference on Computer Vision, 670–677, 2001.
Crossref
[15]
Tappen, M. F.; Adelson, E. H.; Freeman, W. T. Estimating intrinsic component images using non-linear regression. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 1992–1999, 2006.
Crossref
[16]
Tappen, M.; Freeman, W. T.; Adelson, E. H. Recovering intrinsic images from a single image. In: Proceedings of the Advances in Neural Information Processing Systems, 2002.
[17]

Zhao, Q.; Tan, P.; Dai, Q.; Shen, L.; Wu, E.; Lin, S. A closed-form solution to retinex with nonlocal texture constraints. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 34, No. 7, 1437–1444, 2012.
Crossref Google Scholar
[18]
Shen, L.; Yeo, C. Intrinsic images decomposition using a local and global sparse representation of reflectance. In: Proceedings of the Conference on Computer Vision and Pattern Recognition, 697–704, 2011.
Crossref
[19]
Rother, C.; Kiefel, M.; Zhang, L.; Schölkopf, B.; Gehler, P. V. Recovering intrinsic images with a global sparsity prior on reflectance. In: Proceedings of the Advances in Neural Information Processing Systems, 2011.
[20]

Garces, E.; Munoz, A.; Lopez-Moreno, J.; Gutierrez, D. Intrinsic images by clustering. Computer Graphics Forum Vol. 31, No. 4, 1415–1424, 2012.
Crossref Google Scholar
[21]
Barron, J. T.; Malik, J. Color constancy, intrinsic images, and shape estimation, In: Computer Vision – ECCV 2012. Lecture Notes in Computer Science, Vol. 7575. Fitzgibbon, A.; Lazebnik, S.; Perona, P.; Sato, Y.; Schmid, C. Eds. Springer Cham, 57–70, 2012.
Crossref
[22]

Bell, S.; Bala, K.; Snavely, N. Intrinsic images in the wild. ACM Transactions on Graphics Vol. 33, No. 4, Article No. 159, 2014.
Crossref Google Scholar
[23]

Xiao, Y. P.; Lai, Y. K.; Zhang, F. L.; Li, C.; Gao, L. A survey on deep geometry learning: From a representation perspective. Computational Visual Media Vol. 6, No. 2, 113–133, 2020.
Crossref Google Scholar
[24]

Huo, Y.; Yoon, S. E. A survey on deep learning-based Monte Carlo denoising. Computational Visual Media Vol. 7, No. 2, 169–185, 2021.
Crossref Google Scholar
[25]

Guo, M. H.; Xu, T. X.; Liu, J. J.; Liu, Z. N.; Jiang, P. T.; Mu, T. J.; Zhang, S. H.; Martin, R. R.; Cheng, M. M.; Hu, S. M. Attention mechanisms in computer vision: A survey. Computational Visual Media Vol. 8, No. 3, 331–368, 2022.
Crossref Google Scholar
[26]
Li, Z.; Snavely, N. CGIntrinsics: Better intrinsic image decomposition through physically-based rendering. In: Computer Vision – ECCV 2018. Lecture Notes in Computer Science, Vol. 11207. Ferrari, V.; Hebert, M.; Sminchisescu, C.; Weiss, Y. Eds. Springer Cham, 371–387, 2018.
Crossref
[27]
Li, Z.; Snavely, N. Learning intrinsic image decomposition from watching the world. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9039–9048, 2018.
Crossref
[28]
Liu, Y.; Li, Y.; You, S.; Lu, F. Unsupervised learning for intrinsic image decomposition from a single image. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3248–3257, 2020.
Crossref
[29]
Zhou, H.; Yu, X.; Jacobs, D. GLoSH: Global–local spherical harmonics for intrinsic image decomposition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 7820–7829, 2019.
Crossref
[30]
Yu, Y.; Smith, W. A. P. InverseRenderNet: Learning single image inverse rendering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3155–3164, 2019.
Crossref
[31]

Land, E.; McCann, J. Lightness and retinex theory. Journal of the Optical Society of America Vol. 61, No. 1, 1–11, 1971.
Crossref Google Scholar
[32]
Chen, Q.; Koltun, V. A simple model for intrinsic image decomposition with depth cues. In: Proceedings of the IEEE International Conference on Computer Vision, 241–248, 2013.
Crossref
[33]
Jeon, J.; Cho, S.; Tong, X.; Lee, S. Intrinsic image decomposition using structure–texture separation and surface normals. In: Computer Vision – ECCV 2014. Lecture Notes in Computer Science, Vol. 8695. Fleet, D.; Pajdla, T.; Schiele, B.; Tuytelaars, T. Eds. Springer Cham, 218–233, 2014.
Crossref
[34]
Barron, J. T.; Malik, J. Intrinsic scene properties from a single RGB-D image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 17–24, 2013.
Crossref
[35]
Lee, K. J.; Zhao, Q.; Tong, X.; Gong, M.; Izadi, S.; Lee, S. U.; Tan, P.; Lin, S. Estimation of intrinsic image sequences from image+depth video. In: Computer Vision – ECCV 2012. Lecture Notes in Computer Science, Vol. 7577. Fitzgibbon, A.; Lazebnik, S.; Perona, P.; Sato, Y.; Schmid, C. Eds. Springer Cham, 327–340, 2012.
Crossref
[36]

Wu, X.; Sahoo, D.; Hoi, S. C. H. Recent advances in deep learning for object detection. Neurocomputing Vol. 396, 39–64, 2020.
Crossref Google Scholar
[37]

Asgari Taghanaki, S.; Abhishek, K.; Cohen, J. P.; Cohen-Adad, J.; Hamarneh, G. Deep semantic segmentation of natural and medical images: A review. Artificial Intelligence Review Vol. 54, No. 1, 137–178, 2021.
Crossref Google Scholar
[38]

Ciaparrone, G.; Luque Sánchez, F.; Tabik, S.; Troiano, L.; Tagliaferri, R.; Herrera, F. Deep learning in video multi-object tracking: A survey. Neurocomputing Vol. 381, 61–88, 2020.
Crossref Google Scholar
[39]

Anwar, S.; Khan, S.; Barnes, N. A deep journey into super-resolution. ACM Computing Surveys Vol. 53, No. 3, Article No. 60, 2021.
Crossref Google Scholar
[40]
Narihira, T.; Maire, M.; Yu, S. X. Direct intrinsics: Learning albedo-shading decomposition by convolutional regression. In: Proceedings of the IEEE International Conference on Computer Vision, 2992–2992, 2015.
Crossref
[41]
Eigen, D.; Puhrsch, C.; Fergus, R. Depth map prediction from a single image using a multi-scale deep network. arXiv preprint arXiv: 1406.2283, 2014.
[42]
Butler, D. J.; Wulff, J.; Stanley, G.; Black, M. J. A naturalistic open source movie for optical flow evaluation. In: Proceedings of the European Conference on Computer Vision, 611–625, 2015.
Crossref
[43]
Grosse, R.; Johnson, M. K.; Adelson, E. H.; Freeman, W. T. Ground truth dataset and baseline evaluations for intrinsic image algorithms. In: Proceedings of the IEEE 12th International Conference on Computer Vision, 2335–2342, 2009.
Crossref
[44]
Fan, Q.; Yang, J.; Hua, G.; Chen, B.; Wipf, D. Revisiting deep intrinsic image decompositions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8944–8952, 2018.
Crossref
[45]
Qian, Y.; Shi, M.; Kamarainen, J. K.; Matas, J. Fast Fourier intrinsic network. In: Proceedings of the IEEE Winter Conference on Applications of Computer Vision, 3168–3178, 2021.
Crossref
[46]
Das, P.; Karaoglu, S.; Gevers, T. PIE-net: Photometric invariant edge guided network for intrinsic image decomposition. In: Proceedings of the IEEE/ CVF Conference on Computer Vision and Pattern Recognition, 19790–19799, 2022.
Crossref
[47]
Jakob, W. Mitsuba physically-based renderer. 2010. Available at https://www.mitsuba-renderer.org/index_old.html
[48]
Janner, M.; Wu, J.; Kulkarni, T. D.; Yildirim, I.; Tenenbaum, J. Selfsupervised intrinsic image decomposition. In: Proceedings of the 31st Neural Information Processing Systems, 2017.
[49]
Ma, W.-C.; Chu, H.; Zhou, B.; Urtasun, R.; Torralba, A. Single image intrinsic decomposition without a single intrinsic image. In: Proceedings of the European Conference on computer vision, 201–217, 2018.
Crossref
[50]
Nestmeyer, T.; Gehler, P. V. Reflectance adaptive filtering improves intrinsic image estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 6789–6798, 2017.
Crossref
[51]
Baslamisli, A. S.; Le, H. A.; Gevers, T. CNN based learning using reflection and retinex models for intrinsic image decomposition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 6674–6683, 2018.
Crossref
[52]
Hui, Z.; Chakrabarti, A.; Sunkavalli, K.; Sankaranarayanan, A. C. Learning to separate multiple illuminants in a single image. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3780–3789, 2019.
Crossref
[53]
Li, C.; Zhou, K.; Lin, S. Simulating makeup through physics-based manipulation of intrinsic image layers. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4621–4629, 2015.
Crossref
[54]

Karsch, K.; Hedau, V.; Forsyth, D.; Hoiem, D. Rendering synthetic objects into legacy photographs. ACM Transactions on Graphics Vol. 30, No. 6, 1–12, 2011.
Crossref Google Scholar
[55]

Karsch, K.; Sunkavalli, K.; Hadap, S.; Carr, N.; Jin H.; Fonte, R.; Sittig, M.; Forsyth, D. Automatic scene inference for 3D object compositing. ACM Transactions on Graphics Vol. 33, No. 3, Article No. 32, 2014.
Crossref Google Scholar
[56]

Shekhar, S.; Reimann, M.; Mayer, M.; Semmo, A.; Pasewaldt, S.; Döllner, J.; Trapp, M. Interactive photo editing on smartphones via intrinsic decomposition. Computer Graphics Forum Vol. 40, No. 2, 497–510, 2021.
Crossref Google Scholar
[57]
He, K.; Zhang, X.; Ren, S.; Sun, J. Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 770–778, 2016.
Crossref
[58]
Hu, J.; Shen, L.; Sun, G. Squeeze-and-excitation networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 7132–7141, 2018.
Crossref
[59]
Kovacs, B.; Bell, S.; Snavely, N.; Bala, K. Shading annotations in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 6998–7007, 2017.
Crossref
[60]

Wang X.; Liang X.; Yang B.; Li, F. W. B. No-reference synthetic image quality assessment with convolutional neural network and local image saliency. Computational Visual Media Vol. 5, No. 2, 193–208, 2019.
Crossref Google Scholar
[61]
Wang, Z.; Simoncelli, E. P.; Bovik, A. C. Multiscale structural similarity for image quality assessment. In: Proceedings of the 37th Asilomar Conference on Signals, Systems & Computers, 1398–1402, 2003.
[62]
Wang, L. A survey on IQA. arXiv preprint arXiv: 2109.00347, 2021.
[63]

Lettry, L.; Vanhoey, K.; van Gool, L. Unsupervised deep single-image intrinsic decomposition using illumination-varying image sequences. Computer Graphics Forum Vol. 37, No. 7, 409–419, 2018.
Crossref Google Scholar
[64]
Zhou, T.; Krahenbuhl, P.; Efros, A. A. Learning data-driven reflectance priors for intrinsic image decomposition. In: Proceedings of the IEEE International Conference on Computer Vision, 3469–3477, 2015.
Crossref
[65]
Shi, J.; Dong, Y.; Su, H.; Yu, S. X. Learning non-lambertian object intrinsics across ShapeNet categories. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1685–1694, 2017.
Crossref
[66]
Baslamisli, A. S.; Groenestege, T. T.; Das, P.; Le, H.-A.; Karaoglu, S.; Gevers, T. Joint learning of intrinsic images and semantic segmentation. In: Proceedings of the 15th European Conference on Computer Vision, 286–302, 2018.
Crossref
[67]
Wang, Z.; Lu, F. Single image intrinsic decomposition with discriminative feature encoding. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshop, 4310–4319, 2019.
Crossref
[68]
Jin, Y.; Li, R.; Yang, W.; Tan, R. T. Estimating reflectance layer from A single image: Integrating reflectance guidance and shadow/specular aware learning. arXiv preprint arXiv: 2211.14751, 2022.
[69]
Sengupta, S.; Gu, J.; Kim, K.; Liu, G.; Jacobs, D.; Kautz, J. Neural inverse rendering of an indoor scene from a single image. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 8597–8606, 2019.
Crossref
[70]

Baslamisli, A. S.; Das, P.; Le, H. A.; Karaoglu, S.; Gevers, T. ShadingNet: Image intrinsics by fine-grained shading decomposition. International Journal of Computer Vision Vol. 129, No. 8, 2445–2473, 2021.
Crossref Google Scholar
[71]

Zhang, Q.; Zhou, J.; Zhu, L.; Sun, W.; Xiao, C.; Zheng, W. S. Unsupervised intrinsic image decomposition using internal self-similarity cues. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 44, No. 12, 9669–9686, 2022.
Crossref Google Scholar
[72]
Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; et al. Pytorch: An imperative style, high-performance deep learning library. In: Proceedings of the 33rd Conference on Neural Information Processing Systems, 2019.
[73]
Kingma, D. P.; Ba, J.; Hammad, M. M. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
[74]
Bi, S.; Kalantari, N. K.; Ramamoorthi, R. Deep hybrid real and synthetic training for intrinsic decomposition. arXiv preprint arXiv: 1807.11226, 2018.
Computational Visual Media
Volume 11 Issue 2,
April 2025
Pages 305-325
DOI: 10.26599/CVM.2025.9450378
Cite this article:
Sha H, Han Y, Xiao Y, et al. Lossless intrinsic image decomposition via learning shading feature filtering. Computational Visual Media, 2025, 11(2): 305-325. https://doi.org/10.26599/CVM.2025.9450378

33

Views

1

Downloads

0

Crossref

0

Web of Science

0

Scopus

0

CSCD

Altmetrics
Received: 19 March 2023
Accepted: 01 September 2023
Published: 08 May 2025
© The Author(s) 2025.

This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

To submit a manuscript, please go to https://jcvm.org.
Return