Journal Home > Volume 9 , Issue 4
Computational Visual Media 2023, 9(4): 843-858 https://doi.org/10.1007/s41095-022-0297-1
Research Article | Open Access | Issue | Published: 03 July 2023

Light field super-resolution using complementary-view feature attention

Show Author's Information Wei Zhang1,  Wei Ke1( ),  Da Yang2( ),  Hao Sheng1,2,  Zhang Xiong1,2
Faculty of Applied Sciences, Macao Polytechnic University, Macao SAR 999078, China
State Key Laboratory of Software Development Environment, School of Computer Science and Engineering, Beihang University, Beijing 100191, China, and Beihang Hangzhou Innovation Institute Yuhang, Xixi Octagon City, Yuhang District, Hangzhou 310023, China
Keywords:
attention, super-resolution (SR), light field (LF)
Cite this article:
Zhang W, Ke W, Yang D, et al. Light field super-resolution using complementary-view feature attention. Computational Visual Media, 2023, 9(4): 843-858. https://doi.org/10.1007/s41095-022-0297-1
Download citation
EndNote(RIS)
BibTeX

339

Views

12

Downloads

Citations

6

Crossref

7

WoS

8

Scopus

0

CSCD

Abstract Full text About this article

Light field (LF) cameras record multiple perspectives by a sparse sampling of real scenes, and these perspectives provide complementary information. This information is beneficial to LF super-resolution (LFSR). Compared with traditional single-imagesuper-resolution, LF can exploit parallax structure and perspective correlation among different LF views. Furthermore, the performance of existing methods are limited as they fail to deeply explore the complementary information across LF views. In this paper, we propose a novel network, called the light field complementary-view feature attention network (LF-CFANet), to improve LFSR by dynamically learning the complementary information in LF views. Specifically, we design a residual complementary-view spatial and channel attention module (RCSCAM) to effectively interact with complementary information between complementary views. Moreover, RCSCAM captures the relationships between different channels, and it is able to generate informative features for reconstructing LF images while ignoring redundant information. Then, a maximum-difference information supplementary branch (MDISB) is used to supplement information from the maximum-difference angular positions based on the geometric structure of LF images. This branch also can guide the process of reconstruction. Experimental results on both synthetic and real-world datasets demonstrate the superiority of our method. The proposed LF-CFANet has a more advanced reconstruction performance that displays faithful details with higher SR accuracy than state-of-the-art methods.


menu
Abstract
Full text
Outline
About this article

Light field super-resolution using complementary-view feature attention

Show Author's information Wei Zhang1, Wei Ke1( ), Da Yang2( ), Hao Sheng1,2, Zhang Xiong1,2
Faculty of Applied Sciences, Macao Polytechnic University, Macao SAR 999078, China
State Key Laboratory of Software Development Environment, School of Computer Science and Engineering, Beihang University, Beijing 100191, China, and Beihang Hangzhou Innovation Institute Yuhang, Xixi Octagon City, Yuhang District, Hangzhou 310023, China

Abstract

Light field (LF) cameras record multiple perspectives by a sparse sampling of real scenes, and these perspectives provide complementary information. This information is beneficial to LF super-resolution (LFSR). Compared with traditional single-imagesuper-resolution, LF can exploit parallax structure and perspective correlation among different LF views. Furthermore, the performance of existing methods are limited as they fail to deeply explore the complementary information across LF views. In this paper, we propose a novel network, called the light field complementary-view feature attention network (LF-CFANet), to improve LFSR by dynamically learning the complementary information in LF views. Specifically, we design a residual complementary-view spatial and channel attention module (RCSCAM) to effectively interact with complementary information between complementary views. Moreover, RCSCAM captures the relationships between different channels, and it is able to generate informative features for reconstructing LF images while ignoring redundant information. Then, a maximum-difference information supplementary branch (MDISB) is used to supplement information from the maximum-difference angular positions based on the geometric structure of LF images. This branch also can guide the process of reconstruction. Experimental results on both synthetic and real-world datasets demonstrate the superiority of our method. The proposed LF-CFANet has a more advanced reconstruction performance that displays faithful details with higher SR accuracy than state-of-the-art methods.

Keywords: attention, super-resolution (SR), light field (LF)

References(50)

[1]
Huang, F.-C.; Luebke, D.; Wetzstein, G. The light field stereoscope. In: Proceedings of the ACM SIGGRAPH Emerging Technologies, Article No. 24, 2015.
DOI
[2]
Yu, J. A light-field journey to virtual reality. IEEE MultiMedia Vol. 24, No. 2, 104–112, 2017.
DOI Google Scholar
[3]
Sheng, H.; Wang, S.; Zhang, Y.; Yu, D. X.; Cheng, X. Z.; Lyu, W. F.; Xiong, Z. Near-online tracking with co-occurrence constraints in blockchain-based edge computing. IEEE Internet of Things Journal Vol. 8, No. 4, 2193–2207, 2021.
DOI Google Scholar
[4]
Sheng, H.; Zhang, Y.; Wu, Y. B.; Wang, S.; Lyu, W. F.; Ke, W.; Xiong, Z. Hypothesis testing based tracking with spatio-temporal joint interaction modeling. IEEE Transactions on Circuits and Systems for Video Technology Vol. 30, No. 9, 2971–2983, 2020.
DOI Google Scholar
[5]
Wang, S.; Sheng, H.; Zhang, Y.; Wu, Y. B.; Xiong, Z. A general recurrent tracking framework without real data. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 13199–13208, 2021.
DOI
[6]
Wang, S.; Sheng, H.; Zhang, Y.; Yang, D.; Shen, J.; Chen, R. Blockchain-empowered distributed multi-camera multi-target tracking in edge computing. IEEE Transactions on Industrial Informatics, , 2023.
DOI Google Scholar
[7]
Zhu, H.; Wang, Q.; Yu, J. Y. Occlusion-model guided antiocclusion depth estimation in light field. IEEE Journal of Selected Topics in Signal Processing Vol. 11, No. 7, 965–978, 2017.
DOI Google Scholar
[8]
Piao, Y. R.; Li, X.; Zhang, M.; Yu, J. Y.; Lu, H. C. Saliency detection via depth-induced cellular automata on light field. IEEE Transactions on Image Processing Vol. 29, 1879–1889, 2020.
DOI Google Scholar
[9]
Zhang, M.; Li, J.; Ji, W.; Piao, Y.; Lu, H. Memory-oriented decoder for light field salient object detection. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems, Article No. 81, 898–908, 2019.
[10]
Bishop, T. E.; Favaro, P. The light field camera: Extended depth of field, aliasing, and superresolution. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 34, No. 5, 972–986, 2012.
DOI Google Scholar
[11]
Wanner, S.; Goldluecke, B. Spatial and angular variational super-resolution of 4D light fields. In: Computer Vision – ECCV 2012. Lecture Notes in Computer Science, Vol. 7576. Fitzgibbon, A.; Lazebnik, S.; Perona, P.; Sato, Y.; Schmid, C. Eds. Springer Berlin, Heidelberg, 608–621, 2012.
DOI
[12]
Wanner, S.; Goldluecke, B. Variational light field analysis for disparity estimation and super-resolution. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 36, No. 3, 606–619, 2014.
DOI Google Scholar
[13]
Mitra, K.; Veeraraghavan, A. Light field denoising, light field superresolution and stereo camera based refocussing using a GMM light field patch prior. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, 22–28, 2012.
DOI
[14]
Yuan, Y.; Cao, Z. Q.; Su, L. J. Light-field image superresolution using a combined deep CNN based on EPI. IEEE Signal Processing Letters Vol. 25, No. 9, 1359–1363, 2018.
DOI Google Scholar
[15]
Yoon, Y.; Jeon, H. G.; Yoo, D.; Lee, J. Y.; Kweon, I. S. Learning a deep convolutional network for light-field image super-resolution. In: Proceedings of the IEEE International Conference on Computer Vision Workshop, 57–65, 2015.
DOI
[16]
Yoon, Y.; Jeon, H. G.; Yoo, D.; Lee, J. Y.; Kweon, I. S. Light-field image super-resolution using convolutional neural network. IEEE Signal Processing Letters Vol. 24, No. 6, 848–852, 2017.
DOI Google Scholar
[17]
Li, D. L.; Yang, D.; Wang, S. Z.; Sheng, H. Light field super-resolution based on spatial and angular attention. In: Wireless Algorithms, Systems, and Applications. Lecture Notes in Computer Science, Vol. 12937. Liu, Z.; Wu, F.; Das, S. K. Eds. Springer Cham, 314–325, 2021.
DOI
[18]
Zhang, S.; Lin, Y. F.; Sheng, H. Residual networks for light field image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 11038–11047, 2019.
DOI
[19]
Yeung, H. W. F.; Hou, J. H.; Chen, X. M.; Chen, J.; Chen, Z. B.; Chung, Y. Y. Light field spatial super-resolution using deep efficient spatial-angular separable convolution. IEEE Transactions on Image Processing Vol. 28, No. 5, 2319–2330, 2019.
DOI Google Scholar
[20]
Hu, J.; Shen, L.; Sun, G. Squeeze-and-excitation networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 7132–7141, 2018.
DOI
[21]
Wang, L. G.; Wang, Y. Q.; Liang, Z. F.; Lin, Z. P.; Yang, J. G.; An, W.; Guo, Y. L. Learning parallax attention for stereo image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 12242–12251, 2019.
DOI
[22]
Wang, Y. Q.; Ying, X. Y.; Wang, L. G.; Yang, J. G.; An, W.; Guo, Y. L. Symmetric parallax attention for stereo image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 766–775, 2021.
DOI
[23]
Yang, W. M.; Zhang, X. C.; Tian, Y. P.; Wang, W.; Xue, J. H.; Liao, Q. M. Deep learning for single image super-resolution: A brief review. IEEE Transactions on Multimedia Vol. 21, No. 12, 3106–3121, 2019.
DOI Google Scholar
[24]
Wang, Z. H.; Chen, J.; Hoi, S. C. H. Deep learning for image super-resolution: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 43, No. 10, 3365–3387, 2021.
DOI Google Scholar
[25]
Dong, C.; Loy, C. C.; He, K. M.; Tang, X. O. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 38, No. 2, 295–307, 2015.
DOI Google Scholar
[26]
Dong, C.; Loy, C. C.; He, K. M.; Tang, X. O. Learning a deep convolutional network for image super-resolution. In: Computer Vision – ECCV 2014. Lecture Notes in Computer Science, Vol. 8692. Fleet, D.; Pajdla, T.; Schiele, B.; Tuytelaars, T. Eds. Springer Cham, 184–199, 2014.
DOI
[27]
Kim, J.; Lee, J. K.; Lee, K. M. Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1646–1654, 2016.
DOI
[28]
Lim, B.; Son, S.; Kim, H.; Nah, S.; Lee, K. M. Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 1132–1140, 2017.
DOI
[29]
Timofte, R.; Agustsson, E.; Van Gool, L.; Yang, M.-H.; Zhang, L. NTIRE 2017 challenge on single image super-resolution: Methods and results. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 1110–1121, 2017.
DOI
[30]
Zhang, Y. L.; Tian, Y. P.; Kong, Y.; Zhong, B. N.; Fu, Y. Residual dense network for image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2472–2481, 2018.
DOI
[31]
Zhang, Y. L.; Tian, Y. P.; Kong, Y.; Zhong, B. N.; Fu, Y. Residual dense network for image restoration. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 43, No. 7, 2480–2495, 2021.
DOI Google Scholar
[32]
Zhang, Y. L.; Li, K. P.; Li, K.; Wang, L. C.; Zhong, B. N.; Fu, Y. Image super-resolution using very deep residual channel attention networks. In: Computer Vision – ECCV 2018. Lecture Notes in Computer Science, Vol. 11211. Ferrari, V.; Hebert, M.; Sminchisescu, C.; Weiss, Y. Eds. Springer Cham, 294–310, 2018.
DOI
[33]
Dai, T.; Cai, J. R.; Zhang, Y. B.; Xia, S. T.; Zhang, L. Second-order attention network for single image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 11057–11066, 2019.
DOI
[34]
Zhang, F. L.; Wang, J.; Shechtman, E.; Zhou, Z. Y.; Shi, J. X.; Hu, S. M. PlenoPatch: Patch-based plenoptic image manipulation. IEEE Transactions on Visualization and Computer Graphics Vol. 23, No. 5, 1561–1573, 2017.
DOI Google Scholar
[35]
Rossi, M.; Frossard, P. Geometry-consistent light field super-resolution via graph-based regularization. IEEE Transactions on Image Processing Vol. 27, No. 9, 4207–4218, 2018.
DOI Google Scholar
[36]
Alain, M.; Smolic, A. Light field super-resolution via LFBM5D sparse coding. In: Proceedings of the 25th IEEE International Conference on Image Processing, 2501–2505, 2018.
DOI
[37]
Huang, Y.; Wang, W.; Wang, L. Bidirectional recurrent convolutional networks for multi-frame super-resolution. In: Proceedings of the 28th International Conference on Neural Information Processing Systems, 235–243, 2015.
[38]
Wang, Y. L.; Liu, F.; Zhang, K. B.; Hou, G. Q.; Sun, Z. N.; Tan, T. N. LFNet: A novel bidirectional recurrent convolutional neural network for light-field image super-resolution. IEEE Transactions on Image Processing Vol. 27, No. 9, 4274–4286, 2018.
DOI Google Scholar
[39]
Jin, J.; Hou, J. H.; Chen, J.; Kwong, S. Light field spatial super-resolution via deep combinatorialgeometry embedding and structural consistency regularization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2257–2266, 2020.
DOI
[40]
Wang, Y. Q.; Wang, L. G.; Yang, J. G.; An, W.; Yu, J. Y.; Guo, Y. L. Spatial-angular interaction for light field image super-resolution. In: Computer Vision – ECCV 2020. Lecture Notes in Computer Science, Vol. 12368. Vedaldi, A.; Bischof, H.; Brox, T.; Frahm, J. M. Eds. Springer Cham, 290–308, 2020.
DOI
[41]
Mo, Y.; Wang, Y.; Xiao, C.; Yang, J.; An, W. Dense dual-attention network for light field image super-resolution. IEEE Transactions on Circuits and Systems for Video Technology Vol. 32, No. 7, 4431–4443, 2022.
DOI Google Scholar
[42]
Rerabek, M.; Ebrahimi, T. New light field image dataset. In: Proceedings of the 8th International Conference on Quality of Multimedia Experience, 2016.
[43]
Honauer, K.; Johannsen, O.; Kondermann, D.; Goldluecke, B. A dataset and evaluation methodology for depth estimation on 4D light fields. In: Computer Vision – ACCV 2016. Lecture Notes in Computer Science, Vol. 10113. Lai, S. H.; Lepetit, V.; Nishino, K.; Sato, Y. Eds. Springer Cham, 19–34, 2017.
DOI
[44]
Wanner, S.; Meister, S.; Goldluecke, B. Datasets and benchmarks for densely sampled 4D light fields. In: Proceedings of the Vision, Modeling & Visualization, 225–226, 2013.
[45]
Le Pendu, M.; Jiang, X. R.; Guillemot, C. Light field inpainting propagation via low rank matrix completion. IEEE Transactions on Image Processing Vol. 27, No. 4, 1981–1993, 2018.
DOI Google Scholar
[46]
Vaish, V.; Adams, A. The (new) Stanford light field archive. Computer Graphics Laboratory, Stanford University, 2008. Available at http://lightfield.stanford.edu/index.html.
[47]
Raj, A. S.; Lowney, M.; Shah, R.; Wetzstein, G. Stanford Lytro light field archive. 2016. Available at http://lightfields.stanford.edu/LF2016.html.
[48]
Chen, L. C.; Papandreou, G.; Kokkinos, I.; Murphy, K.; Yuille, A. L. DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 40, No. 4, 834–848, 2018.
DOI Google Scholar
[49]
Ying, X. Y.; Wang, Y. Q.; Wang, L. G.; Sheng, W. D.; An, W.; Guo, Y. L. A stereo attention module for stereo image super-resolution. IEEE Signal Processing Letters Vol. 27, 496–500, 2020.
DOI Google Scholar
[50]
Kim, J.; Lee, J. K.; Lee, K. M. Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1646–1654, 2016.
DOI
Publication history
Copyright
Acknowledgements
Rights and permissions

Publication history

Received: 24 February 2022
Accepted: 11 June 2022
Published: 03 July 2023
Issue date: December 2023

Copyright

© The Author(s) 2023.

Acknowledgements

This study was partially supported by the National Key R&D Program of China (2018YFB2100500), the National Natural Science Foundation of China (61872025), the Science and Technology Development Fund, Macau SAR (0001/2018/AFJ), and the Open Fund of the State Key Laboratory of Software Development Environment (SKLSDE-2021ZX-03). We thank the HAWKEYE Group for their support.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduc-tion in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.

Return