Cargando…

Improving Existing Segmentators Performance with Zero-Shot Segmentators

This paper explores the potential of using the SAM (Segment-Anything Model) segmentator to enhance the segmentation capability of known methods. SAM is a promptable segmentation system that offers zero-shot generalization to unfamiliar objects and images, eliminating the need for additional training...

Descripción completa

Detalles Bibliográficos
Autores principales: Nanni, Loris, Fusaro, Daniel, Fantozzi, Carlo, Pretto, Alberto
Formato: Online Artículo Texto
Lenguaje:English
Publicado: MDPI 2023
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10670220/
https://www.ncbi.nlm.nih.gov/pubmed/37998194
http://dx.doi.org/10.3390/e25111502
_version_ 1785139872686669824
author Nanni, Loris
Fusaro, Daniel
Fantozzi, Carlo
Pretto, Alberto
author_facet Nanni, Loris
Fusaro, Daniel
Fantozzi, Carlo
Pretto, Alberto
author_sort Nanni, Loris
collection PubMed
description This paper explores the potential of using the SAM (Segment-Anything Model) segmentator to enhance the segmentation capability of known methods. SAM is a promptable segmentation system that offers zero-shot generalization to unfamiliar objects and images, eliminating the need for additional training. The open-source nature of SAM allows for easy access and implementation. In our experiments, we aim to improve the segmentation performance by providing SAM with checkpoints extracted from the masks produced by mainstream segmentators, and then merging the segmentation masks provided by these two networks. We examine the “oracle” method (as upper bound baseline performance), where segmentation masks are inferred only by SAM with checkpoints extracted from the ground truth. One of the main contributions of this work is the combination (fusion) of the logit segmentation masks produced by the SAM model with the ones provided by specialized segmentation models such as DeepLabv3+ and PVTv2. This combination allows for a consistent improvement in segmentation performance in most of the tested datasets. We exhaustively tested our approach on seven heterogeneous public datasets, obtaining state-of-the-art results in two of them (CAMO and Butterfly) with respect to the current best-performing method with a combination of an ensemble of mainstream segmentator transformers and the SAM segmentator. The results of our study provide valuable insights into the potential of incorporating the SAM segmentator into existing segmentation techniques. We release with this paper the open-source implementation of our method.
format Online
Article
Text
id pubmed-10670220
institution National Center for Biotechnology Information
language English
publishDate 2023
publisher MDPI
record_format MEDLINE/PubMed
spelling pubmed-106702202023-10-30 Improving Existing Segmentators Performance with Zero-Shot Segmentators Nanni, Loris Fusaro, Daniel Fantozzi, Carlo Pretto, Alberto Entropy (Basel) Article This paper explores the potential of using the SAM (Segment-Anything Model) segmentator to enhance the segmentation capability of known methods. SAM is a promptable segmentation system that offers zero-shot generalization to unfamiliar objects and images, eliminating the need for additional training. The open-source nature of SAM allows for easy access and implementation. In our experiments, we aim to improve the segmentation performance by providing SAM with checkpoints extracted from the masks produced by mainstream segmentators, and then merging the segmentation masks provided by these two networks. We examine the “oracle” method (as upper bound baseline performance), where segmentation masks are inferred only by SAM with checkpoints extracted from the ground truth. One of the main contributions of this work is the combination (fusion) of the logit segmentation masks produced by the SAM model with the ones provided by specialized segmentation models such as DeepLabv3+ and PVTv2. This combination allows for a consistent improvement in segmentation performance in most of the tested datasets. We exhaustively tested our approach on seven heterogeneous public datasets, obtaining state-of-the-art results in two of them (CAMO and Butterfly) with respect to the current best-performing method with a combination of an ensemble of mainstream segmentator transformers and the SAM segmentator. The results of our study provide valuable insights into the potential of incorporating the SAM segmentator into existing segmentation techniques. We release with this paper the open-source implementation of our method. MDPI 2023-10-30 /pmc/articles/PMC10670220/ /pubmed/37998194 http://dx.doi.org/10.3390/e25111502 Text en © 2023 by the authors. https://creativecommons.org/licenses/by/4.0/Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
spellingShingle Article
Nanni, Loris
Fusaro, Daniel
Fantozzi, Carlo
Pretto, Alberto
Improving Existing Segmentators Performance with Zero-Shot Segmentators
title Improving Existing Segmentators Performance with Zero-Shot Segmentators
title_full Improving Existing Segmentators Performance with Zero-Shot Segmentators
title_fullStr Improving Existing Segmentators Performance with Zero-Shot Segmentators
title_full_unstemmed Improving Existing Segmentators Performance with Zero-Shot Segmentators
title_short Improving Existing Segmentators Performance with Zero-Shot Segmentators
title_sort improving existing segmentators performance with zero-shot segmentators
topic Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10670220/
https://www.ncbi.nlm.nih.gov/pubmed/37998194
http://dx.doi.org/10.3390/e25111502
work_keys_str_mv AT nanniloris improvingexistingsegmentatorsperformancewithzeroshotsegmentators
AT fusarodaniel improvingexistingsegmentatorsperformancewithzeroshotsegmentators
AT fantozzicarlo improvingexistingsegmentatorsperformancewithzeroshotsegmentators
AT prettoalberto improvingexistingsegmentatorsperformancewithzeroshotsegmentators