Journal Home > Volume 9 , Issue 2
Computational Visual Media 2023, 9(2): 335-350 https://doi.org/10.1007/s41095-022-0277-5
Research Article | Open Access | Issue | Published: 03 January 2023

Deep unfolding multi-scale regularizer network for image denoising

Show Author's Information Jingzhao Xu1,  Mengke Yuan2,3,  Dong-Ming Yan2,3,  Tieru Wu1( )
School of Mathematics, Jilin University, Changchun 130012, China
National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China
Keywords:
deep learning, image denoising, deep unfolding network, multi-scale regularizer
Cite this article:
Xu J, Yuan M, Yan D-M, et al. Deep unfolding multi-scale regularizer network for image denoising. Computational Visual Media, 2023, 9(2): 335-350. https://doi.org/10.1007/s41095-022-0277-5
Download citation
EndNote(RIS)
BibTeX

7276

Views

118

Downloads

Citations

5

Crossref

5

WoS

4

Scopus

0

CSCD

Abstract Full text About this article

Existing deep unfolding methods unroll an optimization algorithm with a fixed number of steps, and utilize convolutional neural networks (CNNs) to learn data-driven priors. However, their performance is limited for two main reasons. Firstly, priors learned in deep feature space need to be converted to the image space at each iteration step, which limits the depth of CNNs and prevents CNNs from exploiting contextual information. Secondly, existing methods only learn deep priors at the single full-resolution scale, so ignore the benefits of multi-scale context in dealing with high level noise. To address these issues, we explicitly consider the image denoising process in the deep feature space and propose the deep unfolding multi-scale regularizer network (DUMRN) for image denoising. The core of DUMRN is the feature-based denoising module (FDM) that directly removes noise in the deep feature space. In each FDM, we construct a multi-scale regularizer block to learn deep prior information from multi-resolution features. We build the DUMRN by stacking a sequence of FDMs and train it in an end-to-end manner. Experimental results on synthetic and real-world benchmarks demonstrate that DUMRN performs favorably compared to state-of-the-art methods.


menu
Abstract
Full text
Outline
About this article

Deep unfolding multi-scale regularizer network for image denoising

Show Author's information Jingzhao Xu1, Mengke Yuan2,3, Dong-Ming Yan2,3, Tieru Wu1( )
School of Mathematics, Jilin University, Changchun 130012, China
National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China

Abstract

Existing deep unfolding methods unroll an optimization algorithm with a fixed number of steps, and utilize convolutional neural networks (CNNs) to learn data-driven priors. However, their performance is limited for two main reasons. Firstly, priors learned in deep feature space need to be converted to the image space at each iteration step, which limits the depth of CNNs and prevents CNNs from exploiting contextual information. Secondly, existing methods only learn deep priors at the single full-resolution scale, so ignore the benefits of multi-scale context in dealing with high level noise. To address these issues, we explicitly consider the image denoising process in the deep feature space and propose the deep unfolding multi-scale regularizer network (DUMRN) for image denoising. The core of DUMRN is the feature-based denoising module (FDM) that directly removes noise in the deep feature space. In each FDM, we construct a multi-scale regularizer block to learn deep prior information from multi-resolution features. We build the DUMRN by stacking a sequence of FDMs and train it in an end-to-end manner. Experimental results on synthetic and real-world benchmarks demonstrate that DUMRN performs favorably compared to state-of-the-art methods.

Keywords: deep learning, image denoising, deep unfolding network, multi-scale regularizer

References(65)

[1]
Elad, M.; Aharon, M. Image denoising via sparse and redundant representations over learned dictionaries. IEEE Transactions on Image Processing Vol. 15, No. 12, 3736–3745, 2006.
DOI Google Scholar
[2]
Mairal, J.; Bach, F.; Ponce, J.; Sapiro, G.; Zisserman, A. Non-local sparse models for image restoration. In: Proceedings of the IEEE 12th International Conference on Computer Vision, 2272–2279, 2019.
[3]
Dong, W. S.; Zhang, L.; Shi, G. M.; Li, X. Nonlocally centralized sparse representation for image restoration. IEEE Transactions on Image Processing Vol. 22, No. 4, 1620–1630, 2013.
DOI Google Scholar
[4]
Buades, A.; Coll, B.; Morel, J. M. A non-local algorithm for image denoising. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 60–65, 2005.
[5]
Buades, A.; Coll, B.; Morel, J. M. Nonlocal image and movie denoising. International Journal of Computer Vision Vol. 76, No. 2, 123–139, 2008.
DOI Google Scholar
[6]
Dabov, K.; Foi, A.; Katkovnik, V.; Egiazarian, K. Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Transactions on Image Processing Vol. 16, No. 8, 2080–2095, 2007.
DOI Google Scholar
[7]
Wu, X. C.; Zhou, B. Y.; Ren, Q. Y.; Guo, W. Multispectral image denoising using sparse and graph Laplacian Tucker decomposition. Computational Visual Media Vol. 6, No. 3, 319–331, 2020.
DOI Google Scholar
[8]
Xie Q.; Zhao Q.; Xu Z. B.; Meng D. Y. Color and direction-invariant nonlocal self-similarity prior and its application to color image denoising. Science China Information Sciences Vol. 63, Article No. 222101, 2020.
DOI Google Scholar
[9]
Dong, W. S.; Shi, G. M.; Li, X. Nonlocal image restoration with bilateral variance estimation: A low-rank approach. IEEE Transactions on Image Processing Vol. 22, No. 2, 700–711, 2013.
DOI Google Scholar
[10]
Gu, S. H.; Xie, Q.; Meng, D. Y.; Zuo, W. M.; Feng, X. C.; Zhang, L. Weighted nuclear norm minimization and its applications to low level vision. International Journal of Computer Vision Vol. 121, No. 2, 183–208, 2017.
DOI Google Scholar
[11]
Lan, X. Y.; Roth, S.; Huttenlocher, D.; Black, M. J. Efficient belief propagation with learned higher-order Markov random fields. In: Computer Vision – ECCV 2006. Lecture Notes in Computer Science, Vol. 3952. Leonardis, A.; Bischof, H.; Pinz, A. Eds. Springer Berlin Heidelberg, 269–282, 2006.
DOI
[12]
Roth, S.; Black, M. J. Fields of experts. International Journal of Computer Vision Vol. 82, No. 2, 205–229, 2009.
DOI Google Scholar
[13]
Chen, F.; Zhang, L.; Yu, H. M. External patch prior guided internal clustering for image denoising. In: Proceedings of the IEEE International Conference on Computer Vision, 603–611, 2015.
DOI
[14]
Zhang, K.; Zuo, W. M.; Chen, Y. J.; Meng, D. Y.; Zhang, L. Beyond a Gaussian denoiser: Residual learning of deep CNN for image denoising. IEEE Transactions on Image Processing Vol. 26, No. 7, 3142–3155, 2017.
DOI Google Scholar
[15]
Zhang K.; Zuo W. M.; Zhang L. FFDNet: Toward a fast and flexible solution for CNN based image denoising. IEEE Transactions on Image Processing Vol. 27, No. 9, 4608–4622, 2018.
DOI Google Scholar
[16]
Chang, M.; Li, Q.; Feng, H.; Xu, Z. Spatial-adaptive network for single image denoising. In: Computer Vision – ECCV 2020. Lecture Notes in Computer Science, Vol. 12375. Vedaldi, A.; Bischof, H.; Brox, T.; Frahm, J. M. Eds. Springer Cham, 171–187, 2020.
[17]
Zhang, Y. L.; Tian, Y. P.; Kong, Y.; Zhong, B. N.; Fu, Y. Residual dense network for image restoration. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 43, No. 7, 2480–2495, 2021.
DOI Google Scholar
[18]
Mou, C.; Zhang, J.; Fan, X. P.; Liu, H. F.; Wang, R. G. COLA-net: Collaborative attention network for image restoration. IEEE Transactions on Multimedia Vol. 24, 1366–1377, 2022.
DOI Google Scholar
[19]
Chen, Y. J.; Pock, T. Trainable nonlinear reaction diffusion: A flexible framework for fast and effective image restoration. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 39, No. 6, 1256–1272, 2017.
DOI Google Scholar
[20]
Gong, D.; Zhang, Z.; Shi, Q. F.; van den Hengel, A.; Shen, C. H.; Zhang, Y. N. Learning deep gradient descent optimization for image deconvolution. IEEE Transactions on Neural Networks and Learning Systems Vol. 31, No. 12, 5468–5482, 2020.
DOI Google Scholar
[21]
Yang, Y.; Sun, J.; Li, H. B.; Xu, Z. B. Deep ADMM-Net for compressive sensing MRI. In: Proceedings of the 30th International Conference on Neural Information Processing Systems, 10–18, 2016.
[22]
Adler, J.; Öktem, O. Learned primal-dual reconstruction. IEEE Transactions on Medical Imaging Vol. 37, No. 6, 1322–1332, 2018.
DOI Google Scholar
[23]
Feng, W. S.; Qiao, P.; Xi, X. Y.; Chen, Y. J. Image denoising via multiscale nonlinear diffusion models. SIAM Journal on Imaging Sciences Vol. 10, No. 3, 1234–1257, 2017.
DOI Google Scholar
[24]
He, K. M.; Zhang, X. Y.; Ren, S. Q.; Sun, J. Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 770–778, 2016.
[25]
Ioffe, S.; Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: Proceedings of the 32nd International Conference on International Conference on Machine Learning, Vol. 37, 448–456, 2015.
[26]
Huang, G.; Liu, Z.; van der Maaten, L.; Weinberger, K. Q. Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2261–2269, 2017.
DOI
[27]
Guo, S.; Yan, Z. F.; Zhang, K.; Zuo, W. M.; Zhang, L. Toward convolutional blind denoising of real photographs. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1712–1722, 2019.
DOI
[28]
Kim, Y.; Soh, J. W.; Park, G. Y.; Cho, N. I. Transfer learning from synthetic to real-noise denoising with adaptive instance normalization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3479–3489, 2020.
DOI
[29]
Yu, K.; Wang, X. T.; Dong, C.; Tang, X. O.; Loy, C. C. Path-restore: Learning network path selection for image restoration. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 44, No. 10, 7078–7092, 2022.
Google Scholar
[30]
Gu, S. H.; Li, Y. W.; van Gool, L.; Timofte, R. Self-guided network for fast image denoising. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2511–2520, 2019.
[31]
Zamir, S. W.; Arora, A.; Khan, S.; Hayat, M.; Khan, F. S.; Yang, M. H.; Shao, L. Learning enriched features for real image restoration and enhancement. In: Computer Vision – ECCV 2020. Lecture Notes in Computer Science, Vol. 12370. Vedaldi, A.; Bischof, H.; Brox, T.; Frahm, J. M. Eds. Springer Cham, 492–511, 2020.
DOI
[32]
Yu, X. J.; Fu, Z. X.; Ge, C. K. A multi-scale generative adversarial network for real-world image denoising. Signal Image Video Processing Vol. 16, No. 1, 257–264, 2022.
DOI Google Scholar
[33]
Ronneberger, O.; Fischer, P.; Brox, T. U-Net: Convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015. Lecture Notes in Computer Science, Vol. 9351. Navab, N.; Hornegger, J.; Wells, W.; Frangi, A. Eds. Springer Cham, 234–241, 2015.
DOI
[34]
Barbu, A. Training an active random field for real-time image denoising. IEEE Transactions on Image Processing Vol. 18, No. 11, 2451–2462, 2009.
DOI Google Scholar
[35]
Sun, J.; Tappen, M. F. Learning non-local range Markov Random field for image restoration. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2745–2752, 2011.
DOI
[36]
Simon, D.; Elad, M. Rethinking the CSC model for natural images. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems, 2274–2284, 2019.
[37]
Schmidt, U.; Roth, S. Shrinkage fields for effective image restoration. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2774–2781, 2014.
DOI
[38]
Zhang, K.; van Gool, L.; Timofte, R. Deep unfolding network for image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3214–3223, 2020.
DOI
[39]
Ren, C.; He, X. H.; Wang, C. C.; Zhao, Z. B. Adaptive consistency prior based deep network for image denoising. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8592–8602, 2021.
DOI
[40]
Qian, N. On the momentum term in gradient descent learning algorithms. Neural Networks Vol. 12, No. 1, 145–151, 1999.
DOI Google Scholar
[41]
Sutskever, I.; Martens, J.; Dahl, G.; Hinton, G. On the importance of initialization and momentum in deep learning. In: Proceedings of the 30th International Conference on International Conference on Machine Learning, Vol. 28, III-1139–III-1147, 2013.
[42]
Lim, B.; Son, S.; Kim, H.; Nah, S.; Lee, K. M. Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 1132–1140, 2017.
DOI
[43]
Nah, S.; Kim, T. H.; Lee, K. M. Deep multi-scale convolutional neural network for dynamic scene deblurring. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 257–265, 2017.
DOI
[44]
Nair, V.; Hinton, G. E. Rectified linear units improve restricted Boltzmann machines. In: Proceedings of the 27th International Conference on International Conference on Machine Learning, 807–814, 2010.
[45]
Zontak, M.; Mosseri, I.; Irani, M. Separating signal from noise using patch recurrence across scales. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1195–1202, 2013.
DOI
[46]
Dong, H.; Pan, J. S.; Xiang, L.; Hu, Z.; Zhang, X. Y.; Wang, F.; Yang, M. H. Multi-scale boosted dehazing network with dense feature fusion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2154–2164, 2020.
DOI
[47]
Irani, M.; Peleg, S. Improving resolution by image registration. CVGIP: Graphical Models and Image Processing Vol. 53, No. 3, 231–239, 1991.
DOI Google Scholar
[48]
Haris, M.; Shakhnarovich, G.; Ukita, N. Deep back-projecti networks for single image super-resolution. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 43, No. 12, 4323–4337, 2021.
DOI Google Scholar
[49]
Dai, S. Y.; Han, M.; Wu, Y.; Gong, Y. H. Bilateral back-projection for single image super resolution. In: Proceedings of the IEEE International Conference on Multimedia and Expo, 1039–1042, 2007.
DOI
[50]
Agustsson, E.; Timofte, R. NTIRE 2017 challenge on single image super-resolution: Dataset and study. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 1122–1131, 2017.
DOI
[51]
Abdelhamed, A.; Lin, S.; Brown, M. S. A high-quality denoising dataset for smartphone cameras. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1692–1700, 2018.
DOI
[52]
Anaya, J.; Barbu, A. RENOIR - A dataset for real low-light image noise reduction. Journal of Visual Communication and Image Representation Vol. 51, 144–154, 2018.
DOI Google Scholar
[53]
Martin, D.; Fowlkes, C.; Tal, D.; Malik, J. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proceedings of the 8th IEEE International Conference on Computer Vision, 416–423, 2001.
[54]
Huang, J. B.; Singh, A.; Ahuja, N. Single image super-resolution from transformed self-exemplars. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 5197–5206, 2015.
DOI
[55]
Franzen, R. Kodak lossless true color image suite. 1999. Available at http://r0k.us/graphics/kodak.
[56]
Plötz, T.; Roth, S. Benchmarking denoising algorithms with real photographs. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2750–2759, 2017.
DOI
[57]
Wang, Z.; Bovik, A. C.; Sheikh, H. R.; Simoncelli, E. P. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing Vol. 13, No. 4, 600–612, 2004.
DOI Google Scholar
[58]
Liu, D.; Wen, B. H.; Fan, Y. C.; Loy, C. C.; Huang, T. S. Non-local recurrent network for image restoration. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems, 1680–1689, 2018.
[59]
Tian, C. W.; Xu, Y.; Zuo, W. M.; Du, B.; Lin, C. W.; Zhang, D. Designing and training of a dual CNN for image denoising. Knowledge-Based Systems Vol. 226, 106949, 2021.
DOI Google Scholar
[60]
Dabov, K.; Foi, A.; Katkovnik, V.; Egiazarian, K. Color image denoising via sparse 3D collaborative filtering with grouping constraint in luminance-chrominance space. In: Proceedings of the IEEE International Conference on Image Processing, 313–316, 2007.
DOI
[61]
Huang, T.; Li, S. J.; Jia, X.; Lu, H. C.; Liu, J. Z. Neighbor2Neighbor: Self-supervised denoising from single noisy images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 14776–14785, 2021.
DOI
[62]
Kingma, D. P.; Ba, J. Adam: A method for stochastic optimization. In: Proceedings of the 3rd International Conference for Learning Representations, 2015.
[63]
Glorot, X.; Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the 13th International Conference on Artificial Intelligence and Statistics, 249–256, 2010.
[64]
Anwar, S.; Barnes, N.; Petersson, L. Attention-based real image restoration. IEEE Transactions on Neural Networks and Learning Systems , 2021.
DOI Google Scholar
[65]
Liu, C.; Yuen, J.; Torralba, A. SIFT flow: Dense correspondence across scenes and its applications. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 33, No. 5, 978–994, 2011.
DOI Google Scholar
Publication history
Copyright
Acknowledgements
Rights and permissions

Publication history

Received: 12 January 2022
Accepted: 22 February 2022
Published: 03 January 2023
Issue date: June 2023

Copyright

© The Author(s) 2022.

Acknowledgements

This work was partially supported by the National Key R&D Program of China (No. 2020YFA0714101) and the National Nature Science Foundation of China (Nos. 61872162, 62102414, 62172415, and 52175493).

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduc-tion in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.

Return