Cargando…

Non-Local and Multi-Scale Mechanisms for Image Inpainting

Recently, deep learning-based techniques have shown great power in image inpainting especially dealing with squared holes. However, they fail to generate plausible results inside the missing regions for irregular and large holes as there is a lack of understanding between missing regions and existin...

Descripción completa

Detalles Bibliográficos
Autores principales: He, Xu, Yin, Yong
Formato: Online Artículo Texto
Lenguaje:English
Publicado: MDPI 2021
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8126100/
https://www.ncbi.nlm.nih.gov/pubmed/34068573
http://dx.doi.org/10.3390/s21093281
_version_ 1783693701448466432
author He, Xu
Yin, Yong
author_facet He, Xu
Yin, Yong
author_sort He, Xu
collection PubMed
description Recently, deep learning-based techniques have shown great power in image inpainting especially dealing with squared holes. However, they fail to generate plausible results inside the missing regions for irregular and large holes as there is a lack of understanding between missing regions and existing counterparts. To overcome this limitation, we combine two non-local mechanisms including a contextual attention module (CAM) and an implicit diversified Markov random fields (ID-MRF) loss with a multi-scale architecture which uses several dense fusion blocks (DFB) based on the dense combination of dilated convolution to guide the generative network to restore discontinuous and continuous large masked areas. To prevent color discrepancies and grid-like artifacts, we apply the ID-MRF loss to improve the visual appearance by comparing similarities of long-distance feature patches. To further capture the long-term relationship of different regions in large missing regions, we introduce the CAM. Although CAM has the ability to create plausible results via reconstructing refined features, it depends on initial predicted results. Hence, we employ the DFB to obtain larger and more effective receptive fields, which benefits to predict more precise and fine-grained information for CAM. Extensive experiments on two widely-used datasets demonstrate that our proposed framework significantly outperforms the state-of-the-art approaches both in quantity and quality.
format Online
Article
Text
id pubmed-8126100
institution National Center for Biotechnology Information
language English
publishDate 2021
publisher MDPI
record_format MEDLINE/PubMed
spelling pubmed-81261002021-05-17 Non-Local and Multi-Scale Mechanisms for Image Inpainting He, Xu Yin, Yong Sensors (Basel) Article Recently, deep learning-based techniques have shown great power in image inpainting especially dealing with squared holes. However, they fail to generate plausible results inside the missing regions for irregular and large holes as there is a lack of understanding between missing regions and existing counterparts. To overcome this limitation, we combine two non-local mechanisms including a contextual attention module (CAM) and an implicit diversified Markov random fields (ID-MRF) loss with a multi-scale architecture which uses several dense fusion blocks (DFB) based on the dense combination of dilated convolution to guide the generative network to restore discontinuous and continuous large masked areas. To prevent color discrepancies and grid-like artifacts, we apply the ID-MRF loss to improve the visual appearance by comparing similarities of long-distance feature patches. To further capture the long-term relationship of different regions in large missing regions, we introduce the CAM. Although CAM has the ability to create plausible results via reconstructing refined features, it depends on initial predicted results. Hence, we employ the DFB to obtain larger and more effective receptive fields, which benefits to predict more precise and fine-grained information for CAM. Extensive experiments on two widely-used datasets demonstrate that our proposed framework significantly outperforms the state-of-the-art approaches both in quantity and quality. MDPI 2021-05-10 /pmc/articles/PMC8126100/ /pubmed/34068573 http://dx.doi.org/10.3390/s21093281 Text en © 2021 by the authors. https://creativecommons.org/licenses/by/4.0/Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
spellingShingle Article
He, Xu
Yin, Yong
Non-Local and Multi-Scale Mechanisms for Image Inpainting
title Non-Local and Multi-Scale Mechanisms for Image Inpainting
title_full Non-Local and Multi-Scale Mechanisms for Image Inpainting
title_fullStr Non-Local and Multi-Scale Mechanisms for Image Inpainting
title_full_unstemmed Non-Local and Multi-Scale Mechanisms for Image Inpainting
title_short Non-Local and Multi-Scale Mechanisms for Image Inpainting
title_sort non-local and multi-scale mechanisms for image inpainting
topic Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8126100/
https://www.ncbi.nlm.nih.gov/pubmed/34068573
http://dx.doi.org/10.3390/s21093281
work_keys_str_mv AT hexu nonlocalandmultiscalemechanismsforimageinpainting
AT yinyong nonlocalandmultiscalemechanismsforimageinpainting