AI Chat Paper
Note: Please note that the following content is generated by AMiner AI. SciOpen does not take any responsibility related to this content.
{{lang === 'zh_CN' ? '文章概述' : 'Summary'}}
{{lang === 'en_US' ? '中' : 'Eng'}}
Chat more with AI
Powered by AMiner. Clicking this button will take you away from SciOpen.
Home Tsinghua Science and Technology Article
PDF (9.1 MB)
Cite
EndNote(RIS) BibTeX
Collect
Submit Manuscript AI Chat Paper
Show Outline
Outline
Show full outline
Hide outline
Outline
Show full outline
Hide outline
Open Access

Grayscale-Assisted RGB Image Conversion from Near-Infrared Images

School of Computer Science & Technology, Beijing Institute of Technology, Beijing 100081, China
RIKEN API, Tokyo 103-0021, Japan, and also with University of Tokyo, Tokyo 103-0027, Japan
Show Author Information

Abstract

Near-InfraRed (NIR) imaging technology plays a pivotal role in assisted driving and safety surveillance systems, yet its monochromatic nature and deficiency in detail limit its further application. Recent methods aim to recover the corresponding RGB image directly from the NIR image using Convolutional Neural Networks (CNN). However, these methods struggle with accurately recovering both luminance and chrominance information and the inherent deficiencies in NIR image details. In this paper, we propose grayscale-assisted RGB image restoration from NIR images to recover luminance and chrominance information in two stages. We address the complex NIR-to-RGB conversion challenge by decoupling it into two separate stages. First, it converts NIR to grayscale images, focusing on luminance learning. Then, it transforms grayscale to RGB images, concentrating on chrominance information. In addition, we incorporate frequency domain learning to shift the image processing from the spatial domain to the frequency domain, facilitating the restoration of the detailed textures often lost in NIR images. Empirical evaluations of our grayscale-assisted framework and existing state-of-the-art methods demonstrate its superior performance and yield more visually appealing results. Code is accessible at: https://github.com/Yiiclass/RING

Keywords

Near-InfraRed (NIR) to RGB conversiongrayscale domain priorimage colorization

References

[1]
K. Enomoto, K. Sakurada, W. Wang, H. Fukui, M. Matsuoka, R. Nakamura, and N. Kawaguchi, Filmy cloud removal on satellite imagery with multispectral conditional generative adversarial nets, in Proc. IEEE Conf. Computer Vision and Pattern Recognition Workshops (CVPRW), Honolulu, HI, USA, 2017, pp. 1533–1541.
Crossref
[2]

L. Cao, D. Huang, Y. Zhang, X. Jiang, and Y. Chen, Brain decoding using fNIRS, Proc. AAAI Conf. Artif. Intell., vol. 35, no. 14, pp. 12602–12611, 2021.
Crossref Google Scholar
[3]
M. T. Chiu, X. Xu, Y. Wei, Z. Huang, A. G. Schwing, R. Brunner, H. Khachatrian, H. Karapetyan, I. Dozier, G. Rose, et al., Agriculture-vision: A large aerial image database for agricultural pattern analysis, in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 2020, pp. 2825–2835.
Crossref
[4]

X. Yuan, S. Guo, C. Li, B. Lu, and S. Lou, Near infrared star centroid detection by area analysis of multi-scale super pixel saliency fusion map, Tsinghua Science and Technology, vol. 24, no. 3, pp. 291–300, 2019.
Crossref Google Scholar
[5]
A. Kumar, A. Gupta, B. Santra, K. S. Lalitha, M. Kolla, M. Gupta, and R. Singh, VPDS: An AI-based automated vehicle occupancy and violation detection system, Proc. AAAI Conf. Artif. Intell., vol. 33, no. 1, pp. 9498–9503, 2019.
Crossref
[6]

C. Li, T. Zhu, L. Liu, X. Si, Z. Fan, and S. Zhai, Cross-modal object tracking: Modality-aware representations and a unified benchmark, Proc. AAAI Conf. Artif. Intell., vol. 36, no. 2, pp. 1289–1296, 2022.
Crossref Google Scholar
[7]
M. Limmer and H. P. A. Lensch, Infrared colorization using deep convolutional neural networks, in Proc. 15th IEEE Int. Conf. Machine Learning and Applications (ICMLA), Anaheim, CA, USA, 2016, pp. 61–68.
Crossref
[8]
A. Mehri and A. D. Sappa, Colorizing near infrared images through a cyclic adversarial approach of unpaired samples, in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition Workshops (CVPRW), Long Beach, CA, USA, 2019, pp. 971–979.
Crossref
[9]
P. L. Suárez, A. D. Sappa, and B. X. Vintimilla, Infrared image colorization based on a triplet DCGAN architecture, in Proc. IEEE Conf. Computer Vision and Pattern Recognition Workshops (CVPRW), Honolulu, HI, USA, 2017, pp. 212–217.
Crossref
[10]

W. Liang, D. Derui, and W. Guoliang, An improved dual GAN for near-infrared image colorization, Infrared Physics & Technology, vol. 116, p. 103764, 2021.
Crossref Google Scholar
[11]

L. Chen, Y. Liu, Y. He, Z. Xie, and X. Sui, Colorization of infrared images based on feature fusion and contrastive learning, Opt. Lasers Eng., vol. 162, p. 107395, 2023.
Crossref Google Scholar
[12]
Z.-Q J. Xu, Y. Zhang, T. Luo, Y. Xiao, and Z. Ma, Frequency principle: Fourier analysis sheds light on deep neural networks, arXiv preprint 1901.06523, 2019.
[13]
L. Liu, Y. Chen, J. Yan, and Y. Zheng, Optimal LED spectral multiplexing for NIR2RGB translation, in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 2022, pp. 12642–12650.
Crossref
[14]

H. Chang, O. Fried, Y. Liu, S. DiVerdi, and A. Finkelstein, Palette-based photo recoloring, ACM Trans. Graph., vol. 34, no. 4, pp. 1–11, 2015.
Crossref Google Scholar
[15]
Z. Xu, T. Wang, F. Fang, Y. Sheng, and G. Zhang, Stylization-based architecture for fast deep exemplar colorization, in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 2020, pp. 9360–9369.
Crossref
[16]

M. He, D. Chen, J. Liao, P. V. Sander, and L. Yuan, Deep exemplar-based colorization, ACM Trans. Graph., vol. 37, no. 4, pp. 1–16, 2018.
Crossref Google Scholar
[17]
Z. Cheng, Q. Yang, and B. Sheng, Deep colorization, in Proc. IEEE Int. Conf. Computer Vision (ICCV), Santiago, Switzzerland, Chile, 2015, pp. 415–423.
Crossref
[18]
R. Zhang, P. Isola, and A. A. Efros, Colorful image colorization. Lecture Notes in Computer Science. Cham, Switzerland: Springer International Publishing, 2016. pp. 649–666,
Crossref
[19]
S. Iizuka, E. Simo-Serra, and H. Ishikawa, Let there be color!: Joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification, ACM Transactions on Graphics, vol. 35, no. 4, pp. 1−11, 2016.
Crossref
[20]

R. Zhang, J.-Y. Zhu, P. Isola, X. Geng, A. S. Lin, T. Yu, and A. A. Efros, Real-time user-guided image colorization with learned deep priors, ACM Trans. Graph., vol. 36, no. 4, pp. 1–11, 2017.
Crossref Google Scholar
[21]
J.-W. Su, H.-K. Chu, and J.-B. Huang, Instance-aware image colorization, in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 2020, pp. 7965–7974.
Crossref
[22]

Q. Liu, Y. Jiang, Z. Tan, D. Chen, Y. Fu, Q. Chu, G. Hua, and N. Yu, Transformer based pluralistic image completion with reduced information loss, IEEE Trans. Pattern Anal. Mach. Intell., doi: 10.1109/TPAMI.2024.3384406.
Crossref Google Scholar
[23]
S. Weng, J. Sun, Y. Li, S. Li, and B. Shi, CT2: Colorization transformer via color tokens, in Proc. 17th European Conf. Computer Vision (ECCV), Tel Aviv, Israel, 2022, pp. 1–16.
Crossref
[24]
X. Guo, Y. Li, and H. Ling, LIME: Low-light image enhancement via illumination map estimation, IEEE Trans. Image Process., vol. 26, no. 2, pp. 982–993, 2017.
Crossref
[25]
F. Lv, F. Lu, J. Wu, and C. Lim, MBLLEN: Low-light image/video enhancement using CNNs, in Proc. of British Machine Vision Conference, Newcastle, UK, 2018, p. 4.
[26]

K. Wei, Y. Fu, Y. Zheng, and J. Yang, Physics-based noise modeling for extreme low-light photography, IEEE Trans. Pattern Anal. Mach. Intell., vol. 44, no. 11, pp. 8520–8537, 2022.
Google Scholar
[27]

M. Li, Y. Fu, T. Zhang, and G. Wen, Supervise-assisted self-supervised deep-learning method for hyperspectral image restoration, IEEE Trans. Neural Netw. Learn. Syst., doi: 10.1109/TNNLS.2024.3386809.
Crossref Google Scholar
[28]

Q. Zhang, Y. Nie, and W.-S. Zheng, Dual illumination estimation for robust exposure correction, Comput. Graph. Forum, vol. 38, no. 7, pp. 243–252, 2019.
Crossref Google Scholar
[29]

L. Chen, Y. Fu, K. Wei, D. Zheng, and F. Heide, Instance segmentation in the dark, Int. J. Comput. Vis., vol. 131, no. 8, pp. 2198–2218, 2023.
Crossref Google Scholar
[30]

T. Zhang, Y. Fu, and J. Zhang, Guided hyperspectral image denoising with realistic data, Int. J. Comput. Vis., vol. 130, no. 11, pp. 2885–2901, 2022.
Crossref Google Scholar
[31]

T. Zhang, Y. Fu, J. Zhang, and C. Yan, Deep guided attention network for joint denoising and demosaicinging real image, Chinese Journal of Electronics, vol. 33, no. 1, pp. 303–312, 2024.
Crossref Google Scholar
[32]
C. Wei, W. Wang, W. Yang, and J. Liu, Deep retinex decomposition for low-light enhancement, in Proc. British Machine Vision Conf., Newcastle, UK, doi: 10.48550/arXiv.1808.04560.
[33]
Y. Jiang, L. Li, J. Zhu, Y. Xue, and H. Ma, DEANet: Decomposition enhancement and adjustment network for low-light image enhancement, Tsinghua Science and Technology, vol. 28, no. 4, pp. 743–753, 2023.
Crossref
[34]

X. Shang, N. An, S. Zhang, and N. Ding, Toward robust and efficient low-light image enhancement: Progressive attentive Retinex architecture search, Tsinghua Science and Technol, vol. 28, no. 3, pp. 580–594, 2023.
Crossref Google Scholar
[35]
P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, Image-to-image translation with conditional adversarial networks, in Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 2017, pp. 5967–5976.
Crossref
[36]
J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, Unpaired image-to-image translation using cycle-consistent adversarial networks, in Proc. IEEE Int. Conf. Computer Vision (ICCV), Venice, Italy, 2017, pp. 2242–2251.
Crossref
[37]
M.-Y. Liu, T. Breuel, and J. Kautz, Unsupervised image-to image translation networks, in Proc. of Conference on Neural Information Processing Systems, Long Beach, CA, USA, 2017, pp. 700−708.
[38]
Z. Yi, H. Zhang, P. Tan, and M. Gong, DualGAN: Unsupervised dual learning for image-to-image translation, in Proc. IEEE Int. Conf. Computer Vision (ICCV), Venice, Italy, 2017, pp. 2868–2876.
Crossref
[39]
B. Li, K. Xue, B. Liu, and Y.-K. Lai, BBDM: Image-to-image translation with Brownian bridge diffusion models, in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), Vancouver, Canada, 2023, pp. 1952–1961.
Crossref
[40]
O. Ronneberger, P. Fischer, and T. Brox, U-net: Convolutional networks for biomedical image segmentation, in Proc. of Medical Image Computing and Computer-assisted Intervention, Munich, Germany, 2015, pp. 234−241.
Crossref
[41]
L. Chi, B. Jiang, and Y. Mu, Fast Fourier convolution, in Proc. Conf. Neural Information Processing Systems, Virtual Event, 2020, pp. 4479−4488.
[42]

E. O. Brigham and R. E. Morrow, The fast Fourier transform, IEEE Spectr., vol. 4, no. 12, pp. 63–70, 1967.
Crossref Google Scholar
[43]
B. Arad and O. Ben-Shahar, Sparse recovery of hyperspectral signal from natural RGB images, in Proc. 14th European Conf. Computer Vision (ECCV), Amsterdam, The Netherlands, 2016, pp. 19–34.
Crossref
[44]
O. Vinyals, C. Blundell, T. Lillicrap, and D. Wierstra, Matching networks for one shot learning, in Proc. of Conference on Neural Information Processing Systems, Barcelona, Spain, 2016. pp. 3637 – 3645.
[45]

Y. Monno, H. Teranaka, K. Yoshizaki, M. Tanaka, and M. Okutomi, Single-sensor RGB-NIR imaging: High-quality system design and prototype implementation, IEEE Sens. J., vol. 19, no. 2, pp. 497–507, 2019.
Crossref Google Scholar
[46]
T. Park, A. A. Efros, R. Zhang, and J.-Y. Zhu, Contrastive learning for unpaired image-to-image translation, in Proc. 16th European Conf. Computer Vision (ECCV), Glasgow, UK, 2020, pp. 319–345.
Crossref
[47]
A. Kirillov, R. Girshick, K. He, and P. Dollár, Panoptic feature pyramid networks, in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 2019, pp. 6392–6401.
Crossref
[48]
K. Chandrasegaran, N.-T. Tran, and N.-M. Cheung, A closer look at Fourier spectrum discrepancies for CNN-generated images detection, in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, 2021, pp. 7196–7205.
Crossref
Tsinghua Science and Technology
Volume 30 Issue 5,
October 2025
Pages 2215-2226
DOI: 10.26599/TST.2024.9010115
Cite this article:
Gao Y, Liu Q, Gu L, et al. Grayscale-Assisted RGB Image Conversion from Near-Infrared Images. Tsinghua Science and Technology, 2025, 30(5): 2215-2226. https://doi.org/10.26599/TST.2024.9010115

223

Views

42

Downloads

0

Crossref

1

Web of Science

0

Scopus

0

CSCD

Altmetrics
Received: 12 April 2024
Revised: 28 May 2024
Accepted: 18 June 2024
Published: 29 April 2025
© The Author(s) 2025.

The articles published in this open access journal are distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/).
Return