Cargando…
Non-Local and Multi-Scale Mechanisms for Image Inpainting
Recently, deep learning-based techniques have shown great power in image inpainting especially dealing with squared holes. However, they fail to generate plausible results inside the missing regions for irregular and large holes as there is a lack of understanding between missing regions and existin...
Autores principales: | , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
MDPI
2021
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8126100/ https://www.ncbi.nlm.nih.gov/pubmed/34068573 http://dx.doi.org/10.3390/s21093281 |
_version_ | 1783693701448466432 |
---|---|
author | He, Xu Yin, Yong |
author_facet | He, Xu Yin, Yong |
author_sort | He, Xu |
collection | PubMed |
description | Recently, deep learning-based techniques have shown great power in image inpainting especially dealing with squared holes. However, they fail to generate plausible results inside the missing regions for irregular and large holes as there is a lack of understanding between missing regions and existing counterparts. To overcome this limitation, we combine two non-local mechanisms including a contextual attention module (CAM) and an implicit diversified Markov random fields (ID-MRF) loss with a multi-scale architecture which uses several dense fusion blocks (DFB) based on the dense combination of dilated convolution to guide the generative network to restore discontinuous and continuous large masked areas. To prevent color discrepancies and grid-like artifacts, we apply the ID-MRF loss to improve the visual appearance by comparing similarities of long-distance feature patches. To further capture the long-term relationship of different regions in large missing regions, we introduce the CAM. Although CAM has the ability to create plausible results via reconstructing refined features, it depends on initial predicted results. Hence, we employ the DFB to obtain larger and more effective receptive fields, which benefits to predict more precise and fine-grained information for CAM. Extensive experiments on two widely-used datasets demonstrate that our proposed framework significantly outperforms the state-of-the-art approaches both in quantity and quality. |
format | Online Article Text |
id | pubmed-8126100 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2021 |
publisher | MDPI |
record_format | MEDLINE/PubMed |
spelling | pubmed-81261002021-05-17 Non-Local and Multi-Scale Mechanisms for Image Inpainting He, Xu Yin, Yong Sensors (Basel) Article Recently, deep learning-based techniques have shown great power in image inpainting especially dealing with squared holes. However, they fail to generate plausible results inside the missing regions for irregular and large holes as there is a lack of understanding between missing regions and existing counterparts. To overcome this limitation, we combine two non-local mechanisms including a contextual attention module (CAM) and an implicit diversified Markov random fields (ID-MRF) loss with a multi-scale architecture which uses several dense fusion blocks (DFB) based on the dense combination of dilated convolution to guide the generative network to restore discontinuous and continuous large masked areas. To prevent color discrepancies and grid-like artifacts, we apply the ID-MRF loss to improve the visual appearance by comparing similarities of long-distance feature patches. To further capture the long-term relationship of different regions in large missing regions, we introduce the CAM. Although CAM has the ability to create plausible results via reconstructing refined features, it depends on initial predicted results. Hence, we employ the DFB to obtain larger and more effective receptive fields, which benefits to predict more precise and fine-grained information for CAM. Extensive experiments on two widely-used datasets demonstrate that our proposed framework significantly outperforms the state-of-the-art approaches both in quantity and quality. MDPI 2021-05-10 /pmc/articles/PMC8126100/ /pubmed/34068573 http://dx.doi.org/10.3390/s21093281 Text en © 2021 by the authors. https://creativecommons.org/licenses/by/4.0/Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/). |
spellingShingle | Article He, Xu Yin, Yong Non-Local and Multi-Scale Mechanisms for Image Inpainting |
title | Non-Local and Multi-Scale Mechanisms for Image Inpainting |
title_full | Non-Local and Multi-Scale Mechanisms for Image Inpainting |
title_fullStr | Non-Local and Multi-Scale Mechanisms for Image Inpainting |
title_full_unstemmed | Non-Local and Multi-Scale Mechanisms for Image Inpainting |
title_short | Non-Local and Multi-Scale Mechanisms for Image Inpainting |
title_sort | non-local and multi-scale mechanisms for image inpainting |
topic | Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8126100/ https://www.ncbi.nlm.nih.gov/pubmed/34068573 http://dx.doi.org/10.3390/s21093281 |
work_keys_str_mv | AT hexu nonlocalandmultiscalemechanismsforimageinpainting AT yinyong nonlocalandmultiscalemechanismsforimageinpainting |