AI Chat Paper
Note: Please note that the following content is generated by AMiner AI. SciOpen does not take any responsibility related to this content.
{{lang === 'zh_CN' ? '文章概述' : 'Summary'}}
{{lang === 'en_US' ? '中' : 'Eng'}}
Chat more with AI
Powered by AMiner. Clicking this button will take you away from SciOpen.
Home Tsinghua Science and Technology Article
PDF (10.6 MB)
Cite
EndNote(RIS) BibTeX
Collect
Submit Manuscript AI Chat Paper
Show Outline
Outline
Show full outline
Hide outline
Outline
Show full outline
Hide outline
Open Access

SemID: Blind Image Inpainting with Semantic Inconsistency Detection

College of Computer Science and Technology, China University of Petroleum (East China), Qingdao 266580, China
CNPC Oriental Geophysical Exploration Co. Ltd., Baoding 072751, China
Show Author Information

Abstract

Most existing image inpainting methods aim to fill in the missing content in the inside-hole region of the target image. However, the areas to be restored in realistically degraded images are unspecified. Previous studies have failed to recover the degradations due to the absence of the explicit mask indication. Meanwhile, inconsistent patterns are blended complexly with the image content. Therefore, estimating whether certain pixels are out of distribution and considering whether the object is consistent with the context is necessary. Motivated by these observations, a two-stage blind image inpainting network, which utilizes global semantic features of the image to locate semantically inconsistent regions and then generates reasonable content in the areas, is proposed. Specifically, the representation differences between inconsistent and available content are first amplified, iteratively predicting the region to be restored from coarse to fine. A confidence-driven inpainting network based on prediction masks is then used to estimate the information regarding missing regions. Furthermore, a multiscale contextual aggregation module is introduced for spatial feature transfer to refine the generated contents. Extensive experiments over multiple datasets demonstrate that the proposed method can generate visually plausible and structurally complete results that are particularly effective in recovering diverse degraded images.

Keywords

blind image inpaintinginconsistent patternrepresentation differencecontextual aggregation

References

[1]

Z. Zou, P. Zhao, and X. Zhao, Automatic segmentation, inpainting, and classification of defective patterns on ancient architecture using multiple deep learning algorithms, Struct. Control. Health Monit., vol. 28, no. 7, p. e2742, 2021.
Crossref Google Scholar
[2]

G. Tudavekar, S. R. Patil, and S. S. Saraf, Dual-tree complex wavelet transform and super-resolution based video inpainting application to object removal and error concealment, CAAI Trans. Intell. Technol., vol. 5, no. 4, pp. 314–319, 2020.
Crossref Google Scholar
[3]
G. Fu, Q. Zhang, L. Zhu, P. Li, and C. Xiao, A multi-task network for joint specular highlight detection and removal, in Proc. 2021 IEEE/CVF Conf. Computer Vision and Pattern Recognition, Nashville, TN, USA, 2021, pp. 7752–7761.
Crossref
[4]
D. Pathak, P. Krähenbühl, J. Donahue, T. Darrell, and A. A. Efros, Context encoders: Feature learning by inpainting, in Proc. 2016 IEEE Conf. Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 2016, pp. 2536–2544.
Crossref
[5]
D. Wang, C. Xie, S. Liu, Z. Niu, and W. Zuo, Image inpainting with edge-guided learnable bidirectional attention maps, arXiv preprint arXiv: 2104.12087, 2021.
[6]
Z. Yi, Q. Tang, S. Azizi, D. Jang, and Z. Xu, Contextual residual aggregation for ultra high-resolution image inpainting, in Proc. 2020 IEEE/CVF Conf. Computer Vision and Pattern Recognition, Seattle, WA, USA, 2020, pp. 7508–7517.
Crossref
[7]
J. Yu, Z. Lin, J. Yang, X. Shen, X. Lu, and T. S. Huang, Generative image inpainting with contextual attention, in Proc. 2018 IEEE/CVF Conf. Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018, pp. 5505–5514.
Crossref
[8]
J. Yu, Z. Lin, J. Yang, X. Shen, X. Lu, and T. S. Huang, Free-form image inpainting with gated convolution, in Proc. 2019 IEEE/CVF Int. Conf. Computer Vision, Seoul, Republic of Korea, 2019, pp. 4471–4480.
Crossref
[9]

Y. Guo and H. Ma, Image blind deblurring using an adaptive patch prior, Tsinghua Science and Technology, vol. 24, no. 2, pp. 238–248, 2019.
Crossref Google Scholar
[10]

N. Cai, Z. Su, Z. Lin, H. Wang, Z. Yang, and B. W. K. Ling, Blind inpainting using the fully convolutional neural network, Vis. Comput., vol. 33, no. 2, pp. 249–261, 2017.
Crossref Google Scholar
[11]
Y. Liu, J. Pan, and Z. Su, Deep blind image inpainting, in Proc. 9 th Int. Conf. Intelligent Science and Big Data Engineering, Nanjing, China, 2019, pp. 128–141.
Crossref
[12]
Y. Wang, Y. C. Chen, X. Tao, and J. Jia, VCNet: A robust approach to blind image inpainting, in Proc. 16 th European Conf. Computer Vision, Glasgow, UK, 2020, pp. 752–768.
Crossref
[13]
X. Bi, Y. Wei, B. Xiao, and W. Li, RRU-Net: The ringed residual U-net for image splicing forgery detection, in Proc. 2019 IEEE/CVF Conf. Computer Vision and Pattern Recognition Workshops, Long Beach, CA, USA, 2019, pp. 30–39.
Crossref
[14]

X. Wu, K. Xu, and P. Hall, A survey of image synthesis and editing with generative adversarial networks, Tsinghua Science and Technology, vol. 22, no. 6, pp. 660–674, 2017.
Crossref Google Scholar
[15]

C. Ballester, M. Bertalmio, V. Caselles, G. Sapiro, and J. Verdera, Filling-in by joint interpolation of vector fields and gray levels, IEEE Trans. Image Process., vol. 10, no. 8, pp. 1200–1211, 2001.
Crossref Google Scholar
[16]

C. Barnes, E. Shechtman, A. Finkelstein, and D. B. Goldman, PatchMatch: A randomized correspondence algorithm for structural image editing, ACM Trans. Graph., vol. 28, no. 3, p. 24, 2009.
Crossref Google Scholar
[17]

S. Darabi, E. Shechtman, C. Barnes, D. B. Goldman, and P. Sen, Image melding: Combining inconsistent images using patch-based synthesis, ACM Trans. Graph., vol. 31, no. 4, p. 82, 2012.
Crossref Google Scholar
[18]
M. Arjovsky, S. Chintala, and L. Bottou, Wasserstein generative adversarial networks, in Proc. 34 th Int. Conf. Machine Learning, Sydney, Australia, 2017, pp. 214–223.
[19]
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, Generative adversarial nets, in Proc. 27 th Int. Conf. Neural Information Processing Systems, Montreal, Canada, 2014, pp. 2672–2680.
[20]

S. Iizuka, E. Simo-Serra, and H. Ishikawa, Globally and locally consistent image completion, ACM Trans. Graph., vol. 36, no. 4, p. 107, 2017.
Crossref Google Scholar
[21]
G. Liu, F. A. Reda, K. J. Shih, T. C. Wang, A. Tao, and B. Catanzaro, Image inpainting for irregular holes using partial convolutions, in Proc. 15 th European Conf. Computer Vision, Munich, Germany, 2018, pp. 85–100.
Crossref
[22]
K. Nazeri, E. Ng, T. Joseph, F. Qureshi, and M. Ebrahimi, EdgeConnect: Structure guided image inpainting using edge prediction, in Proc. 2019 IEEE/CVF Int. Conf. Computer Vision Workshops, Seoul, Republic of Korea, 2019, pp. 3265–3274.
Crossref
[23]

X. Li, H. Zhang, L. Feng, J. Hu, R. Zhang, and Q. Qiao, Edge-aware image outpainting with attentional generative adversarial networks, IET Image Process., vol. 16, no. 7, pp. 1807–1821, 2022.
Crossref Google Scholar
[24]
W. Xiong, J. Yu, Z. Lin, J. Yang, X. Lu, C. Barnes, and J. Luo, Foreground-aware image inpainting, in Proc. 2019 IEEE/CVF Conf. Computer Vision and Pattern Recognition, Long Beach, CA, USA, 2019, pp. 5840–5848.
Crossref
[25]
C. Cao and Y. Fu, Learning a sketch tensor space for image inpainting of man-made scenes, in Proc. 2021 IEEE/CVF Int. Conf. Computer Vision, Montreal, Canada, 2021, pp. 14509–14518.
Crossref
[26]
J. Zhang, L. Niu, D. Yang, L. Kang, Y. Li, W. Zhao, and L. Zhang, GAIN: Gradient augmented inpainting network for irregular holes, in Proc. 27 th ACM Int. Conf. Multimedia, Nice, France, 2019, pp. 1870–1878.
[27]
L. Liao, J. Xiao, Z. Wang, C. W. Lin, and S. Satoh, Image inpainting guided by coherence priors of semantics and textures, in Proc. 2021 IEEE/CVF Conf. Computer Vision and Pattern Recognition, Nashville, TN, USA, 2021, pp. 6539–6548.
Crossref
[28]
Y. Zeng, J. Fu, H. Chao, and B. Guo, Learning pyramid-context encoder network for high-quality image inpainting, in Proc. 2019 IEEE/CVF Conf. Computer Vision and Pattern Recognition, Long Beach, CA, USA, 2019, pp. 1486–1494.
Crossref
[29]
Y. Zeng, Z. Lin, H. Lu, and V. M. Patel, CR-Fill: Generative image inpainting with auxiliary contextual reconstruction, in Proc. 2021 IEEE/CVF Int. Conf. Computer Vision, Montreal, Canada, 2021, pp. 14164–14173.
Crossref
[30]
S. Navasardyan and M. Ohanyan, Image inpainting with onion convolutions, in Proc. 15 th Asian Conf. Computer Vision, Kyoto, Japan, 2020, pp. 3–19.
Crossref
[31]

F. Wu, Y. Kong, W. Dong, and Y. Wu, Gradient-aware blind face inpainting for deep face verification, Neurocomputing, vol. 331, pp. 301–311, 2019.
Crossref Google Scholar
[32]
H. Zhao, Z. Gu, B. Zheng, and H. Zheng, TransCNN-HAE: Transformer-CNN hybrid AutoEncoder for blind image inpainting, in Proc. 30 th ACM Int. Conf. Multimedia, Lisboa, Portugal, 2022, pp. 6813–6821.
Crossref
[33]

M. Hu, J. Yang, N. Ling, Y. Liu, and J. Fan, Lightweight single image deraining algorithm incorporating visual saliency, IET Image Process., vol. 16, no. 12, pp. 3190–3200, 2022.
Crossref Google Scholar
[34]

P. Li, M. Yun, J. Tian, Y. Tang, G. Wang, and C. Wu, Stacked dense networks for single-image snow removal, Neurocomputing, vol. 367, pp. 152–163, 2019.
Crossref Google Scholar
[35]

X. Li, X. Li, X. Zhang, Y. Liu, J. Liang, Z. Guo, and K. Zhai, A method of inpainting moles and acne on the high-resolution face photos, IET Image Process., vol. 15, no. 3, pp. 833–844, 2021.
Crossref Google Scholar
[36]
K. He, X. Zhang, S. Ren, and J. Sun, Deep residual learning for image recognition, in Proc. 2016 IEEE Conf. Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 2016, pp. 770–778.
Crossref
[37]
X. Wang, R. Girshick, A. Gupta, and K. He, Non-local neural networks, in Proc. 2018 IEEE/CVF Conf. Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018, pp. 7794–7803.
Crossref
[38]
L. A. Gatys, A. S. Ecker, and M. Bethge, Image style transfer using convolutional neural networks, in Proc. 2016 IEEE Conf. Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 2016, pp. 2414–2423.
Crossref
[39]
J. Johnson, A. Alahi, and L. Fei-Fei, Perceptual losses for real-time style transfer and super-resolution, in Proc. 14 th European Conf. Computer Vision, Amsterdam, the Netherlands, 2016, pp. 694–711.
Crossref
[40]
Q. Wang, B. Wu, P. Zhu, P. Li, W. Zuo, and Q. Hu, ECA-Net: Efficient channel attention for deep convolutional neural networks, in Proc. 2020 IEEE/CVF Conf. Computer Vision and Pattern Recognition, Seattle, WA, USA, 2020, pp. 11531–11539.
Crossref
[41]
J. Hu, L. Shen, and G. Sun, Squeeze-and-excitation networks, in Proc. 2018 IEEE/CVF Conf. Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018, pp. 7132–7141.
Crossref
[42]

J. Xu, N. Wang, and Y. Wang, Multi-pyramid image spatial structure based on coarse-to-fine pyramid and scale space, CAAI Trans. Intell. Technol., vol. 3, no. 4, pp. 228–234, 2018.
Crossref Google Scholar
[43]

Q. Hua, L. Chen, P. Li, S. Zhao, and Y. Li, A pixel-channel hybrid attention model for image processing, Tsinghua Science and Technology, vol. 27, no. 5, pp. 804–816, 2022.
Crossref Google Scholar
[44]
Y. Wang, X. Tao, X. Qi, X. Shen, and J. Jia, Image inpainting via generative multi-column convolutional neural networks, in Proc. 32 nd Int. Conf. Neural Information Processing Systems, Montréal, Canada, 2018, pp. 329–338.
[45]
I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. Courville, Improved training of wasserstein GANs, in Proc. 31 st Int. Conf. Neural Information Processing Systems, Long Beach, CA, USA, 2017, pp. 5769–5779.
[46]
T. Karras, S. Laine, and T. Aila, A style-based generator architecture for generative adversarial networks, in Proc. 2019 IEEE/CVF Conf. Computer Vision and Pattern Recognition, Long Beach, CA, USA, 2019, pp. 4401–4410.
Crossref
[47]
T. Karras, T. Aila, S. Laine, and J. Lehtinen, Progressive growing of GANs for improved quality, stability, and variation, in Proc. the 6 th Int. Conf. Learning Representations, arXiv preprint arXiv: 1710.10196, 2017.
[48]

C. Doersch, S. Singh, A. Gupta, J. Sivic, and A. A. Efros, What makes Paris look like Paris? ACM Trans. Graph., vol. 31, no. 4, p. 101, 2012.
Crossref Google Scholar
[49]

B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, and A. Torralba, Places: A 10 million image database for scene recognition, IEEE Trans. Pattern Anal. Mach. Intell., vol. 40, no. 6, pp. 1452–1464, 2018.
Crossref Google Scholar
[50]
J. Liang, L. Niu, F. Guo, T. Long, and L. Zhang, Visible watermark removal via self-calibrated localization and background refinement, in Proc. 29 th ACM Int. Conf. Multimedia, Chengdu, China, 2021, pp. 4426–4434.
Crossref
Tsinghua Science and Technology
Volume 29 Issue 4,
August 2024
Pages 1053-1068
DOI: 10.26599/TST.2023.9010079
Cite this article:
Li X, Wang Z, Chen C, et al. SemID: Blind Image Inpainting with Semantic Inconsistency Detection. Tsinghua Science and Technology, 2024, 29(4): 1053-1068. https://doi.org/10.26599/TST.2023.9010079

563

Views

70

Downloads

2

Crossref

3

Web of Science

3

Scopus

0

CSCD

Altmetrics
Received: 10 March 2023
Revised: 04 July 2023
Accepted: 27 July 2023
Published: 09 February 2024
© The Author(s) 2024.

The articles published in this open access journal are distributed under the terms of theCreative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/).
Return