Journal Home > Volume 3 , Issue 4

European Space Aqency (ESA)’s PROBA-V Earth observation (EO) satellite enables us to monitor our planet at a large scale to study the interaction between vegetation and climate, and provides guidance for important decisions on our common global future. However, the interval at which high-resolution images are recorded spans over several days, in contrast to the availability of lower-resolution images which is often daily. We collect an extensive dataset of both high- and low-resolution images taken by PROBA-V instruments during monthly periods to investigate Multi Image Super-resolution, a technique to merge several low-resolution images into one image of higher quality. We propose a convolutional neural network (CNN) that is able to cope with changes in illumination, cloud coverage, and landscape features which are introduced by the fact that the different images are taken over successive satellite passages at the same region. Given a bicubic upscaling of low resolution images taken under optimal conditions, we find the Peak Signal to Noise Ratio of the reconstructed image of the network to be higher for a large majority of different scenes. This shows that applied machine learning has the potential to enhance large amounts of previously collected EO data during multiple satellite passes.


menu
Abstract
Full text
Outline
About this article

Super-resolution of PROBA-V images using convolutional neural networks

Show Author's information Marcus Märtens( )Dario IzzoAndrej KrzicDaniël Cox
European Space Agency, Noordwijk 2001 AZ, the Netherleneds.

Abstract

European Space Aqency (ESA)’s PROBA-V Earth observation (EO) satellite enables us to monitor our planet at a large scale to study the interaction between vegetation and climate, and provides guidance for important decisions on our common global future. However, the interval at which high-resolution images are recorded spans over several days, in contrast to the availability of lower-resolution images which is often daily. We collect an extensive dataset of both high- and low-resolution images taken by PROBA-V instruments during monthly periods to investigate Multi Image Super-resolution, a technique to merge several low-resolution images into one image of higher quality. We propose a convolutional neural network (CNN) that is able to cope with changes in illumination, cloud coverage, and landscape features which are introduced by the fact that the different images are taken over successive satellite passages at the same region. Given a bicubic upscaling of low resolution images taken under optimal conditions, we find the Peak Signal to Noise Ratio of the reconstructed image of the network to be higher for a large majority of different scenes. This shows that applied machine learning has the potential to enhance large amounts of previously collected EO data during multiple satellite passes.

Keywords: convolutional neural network (CNN), deep learning, remote sensing, super-resolution imaging, Earth observation (EO)

References(31)

[1]
Park, S. C., Park, M. K., Kang, M. G.Super-resolution image reconstruction: A technical overview. IEEE Signal Processing Magazine, 2003, 20(3): 21-36.
[2]
Nasrollahi, K., Moeslund, T. B.Super-resolution: a comprehensive survey. Machine Vision and Applications, 2014, 25(6): 1423-1468.
[3]
Latry, C., Delvit, J. M. Staggered arrays for high resolution earth observing systems. In: Proceedings of SPIE 7452, Earth Observing Systems XIV, 2009: 74520O.
DOI
[4]
Li, L., Wang, W., Luo, H., Ying, S.Super-resolution reconstruction of high-resolution satellite ZY-3 TLC images. Sensors, 2017, 17(5): 1062.
[5]
Dierckx, W., Sterckx, S., Benhadj, I., Livens, S., Duhoux, G., van Achteren, T., Francois, M., Mellab, K., Saint, G.PROBA-V mission for global vegetation monitoring: Standard products and image quality. International Journal of Remote Sensing, 2014, 35(7): 2589-2614.
[6]
Yang, C. Y., Ma, C., Yang, M. H. Single-image super-resolution: a benchmark. In: Computer Vision - ECCV 2014. Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T., Eds. Cham: Springer International Publishing, 2014: 372-386.
[7]
Yang, J. C., Wright, J., Huang, T. S., Ma, Y.Image super-resolution via sparse representation. IEEE Transactions on Image Processing, 2010, 19(11): 2861-2873.
[8]
Dong, C., Loy, C. C., He, K. M., Tang, X. O.Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(2): 295-307.
[9]
Kim, J., Lee, J. K., Lee, K. M. Accurate image super-resolution using very deep convolutional networks. In: Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016: 1646-1654.
DOI
[10]
Dong, C., Loy, C. C., Tang, X. O. Accelerating the super-resolution convolutional neural network. In: Computer Vision - ECCV 2016. Leibe, B., Matas, J., Sebe, N., Welling, M., Eds. Cham: Springer International Publishing, 2016: 391-407.
DOI
[11]
Shi, W. Z., Caballero, J., Huszar, F., Totz, J., Aitken, A. P., Bishop, R., Rueckert, D., Wang, Z. H. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016.
DOI
[12]
Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z. H. et al. Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition, 2017.
DOI
[13]
Yeh, R. A., Chen, C., Lim, T. Y., Schwing, A. G., Hasegawa-Johnson, M., Do, M. N. Semantic image inpainting with deep generative models. In: Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition, 2017: 5485-5493.
DOI
[14]
Timofte, R., Gu, S. H., van Gool, L., Zhang, L., Yang, M. H. NTIRE 2018 challenge on single image super-resolution: Methods and results. In: Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2018: 852-863.
[15]
Timofte, R., Agustsson, E., Van Gool, L., Yang, M. H., Zhang, L., Lim, B., Son, S., Kim, H., Nah, S., Lee, K. M., et al. Ntire 2017 challenge on single image super-resolution: Methods and results. In: Proceedings of Computer Vision and Pattern Recognition Workshops, 2017: 1110-1121.
DOI
[16]
He, K. M., Zhang, X. Y., Ren, S. Q., Sun, J. Deep residual learning for image recognition. In: Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016: 770-778.
[17]
Huang, G., Liu, Z., van der Maaten, L., Weinberger, K. Q. Densely connected convolutional networks. In: Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition, 2017: 4700-4708.
DOI
[18]
Wang, Z., Bovik, A. C., Sheikh, H. R., Simoncelli, E. P.Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 2004, 13(4): 600-612.
[19]
Schultz, R. R., Stevenson, R. L.Extraction of high-resolution frames from video sequences. IEEE Transactions on Image Processing, 1996, 5(6): 996-1011.
[20]
Faramarzi, E., Rajan, D., Fernandes, F. C. A., Christensen, M. P.Blind super resolution of real-life video sequences. IEEE Transactions on Image Processing, 2016, 25(4): 1544-1555.
[21]
Takeda, H., Milanfar, P., Protter, M., Elad, M.Super-resolution without explicit subpixel motion estimation. IEEE Transactions on Image Processing, 2009, 18(9): 1958-1975.
[22]
Mitzel, D., Pock, T., Schoenemann, T., Cremers, D. Video super resolution using duality based TV-L1 optical flow. In: Lecture Notes in Computer Science. Denzler, J., Notni, G., Süße, H., Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2009: 432-441.
DOI
[23]
Brodu, N.Super-resolving multiresolution images with band-independent geometry of multispectral pixels. IEEE Transactions on Geoscience and Remote Sensing, 2017, 55(8): 4610-4617.
[24]
Gillespie, A. R., Kahle, A. B., Walker, R. E.Color enhancement of highly correlated images. II. Channel ratio and "chromaticity’’ transformation techniques. Remote Sensing of Environment, 1987, 22(3): 343-365.
[25]
Thomas, C., Ranchin, T., Wald, L., Chanussot, J.Synthesis of multispectral images to high spatial resolution: A critical review of fusion methods based on remote sensing physics. IEEE Transactions on Geoscience and Remote Sensing, 2008, 46(5): 1301-1312.
[26]
Murthy, K., Shearn, M., Smiley, B. D., Chau, A. H., Levine, J., Robinson, M. D. SkySat-1: very high-resolution imagery from a small satellite. In: Proceedings of SPIE 9241, Sensors, Systems, and Next-Generation Satellites XVIII, 2014: 92411E.
DOI
[27]
Wolters, E., Dierckx, W., Iordache, M. D., Swinnen, E. PROBA-V Products User Manual, 2014.
DOI
[28]
Carlson, T. N., Ripley, D. A.On the relation between NDVI, fractional vegetation cover, and leaf area index. Remote Sensing of Environment, 1997, 62(3): 241-252.
[29]
Pettorelli, N., Vik, J. O., Mysterud, A., Gaillard, J. M., Tucker, C. J., Stenseth, N. C.Using the satellite-derived NDVI to assess ecological responses to environmental change. Trends in Ecology & Evolution, 2005, 20(9): 503-510.
[30]
Wilson, A. M., Jetz, W.Remotely sensed high-resolution global cloud dynamics for predicting ecosystem and biodiversity distributions. PLoS Biology, 2016, 14(3): e1002415.
[31]
Sheikh, H. R., Bovik, A. C.Image information and visual quality. IEEE Transactions on Image Processing, 2006, 15(2):430-444.
Publication history
Copyright
Acknowledgements

Publication history

Received: 11 April 2019
Accepted: 29 April 2019
Published: 28 August 2019
Issue date: December 2019

Copyright

© Tsinghua University Press 2019

Acknowledgements

Acknowledgements The authors would like to thank the anonymous reviewers for their extremely detailed comments and critical questions, which helped to make the presentation of these results clearer and increased the overall quality of this work.

Return