Cargando…
SEG: Segmentation Evaluation in absence of Ground truth labels
Identifying individual cells or nuclei is often the first step in the analysis of multiplex tissue imaging (MTI) data. Recent efforts to produce plug-and-play, end-to-end MTI analysis tools such as MCMICRO(1)– though groundbreaking in their usability and extensibility – are often unable to provide u...
Autores principales: | , , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Cold Spring Harbor Laboratory
2023
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9980141/ https://www.ncbi.nlm.nih.gov/pubmed/36865198 http://dx.doi.org/10.1101/2023.02.23.529809 |
_version_ | 1784899856263806976 |
---|---|
author | Sims, Zachary Strgar, Luke Thirumalaisamy, Dharani Heussner, Robert Thibault, Guillaume Chang, Young Hwan |
author_facet | Sims, Zachary Strgar, Luke Thirumalaisamy, Dharani Heussner, Robert Thibault, Guillaume Chang, Young Hwan |
author_sort | Sims, Zachary |
collection | PubMed |
description | Identifying individual cells or nuclei is often the first step in the analysis of multiplex tissue imaging (MTI) data. Recent efforts to produce plug-and-play, end-to-end MTI analysis tools such as MCMICRO(1)– though groundbreaking in their usability and extensibility – are often unable to provide users guidance regarding the most appropriate models for their segmentation task among an endless proliferation of novel segmentation methods. Unfortunately, evaluating segmentation results on a user’s dataset without ground truth labels is either purely subjective or eventually amounts to the task of performing the original, time-intensive annotation. As a consequence, researchers rely on models pre-trained on other large datasets for their unique tasks. Here, we propose a methodological approach for evaluating MTI nuclei segmentation methods in absence of ground truth labels by scoring relatively to a larger ensemble of segmentations. To avoid potential sensitivity to collective bias from the ensemble approach, we refine the ensemble via weighted average across segmentation methods, which we derive from a systematic model ablation study. First, we demonstrate a proof-of-concept and the feasibility of the proposed approach to evaluate segmentation performance in a small dataset with ground truth annotation. To validate the ensemble and demonstrate the importance of our method-specific weighting, we compare the ensemble’s detection and pixel-level predictions – derived without supervision - with the data’s ground truth labels. Second, we apply the methodology to an unlabeled larger tissue microarray (TMA) dataset, which includes a diverse set of breast cancer phenotypes, and provides decision guidelines for the general user to more easily choose the most suitable segmentation methods for their own dataset by systematically evaluating the performance of individual segmentation approaches in the entire dataset. |
format | Online Article Text |
id | pubmed-9980141 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2023 |
publisher | Cold Spring Harbor Laboratory |
record_format | MEDLINE/PubMed |
spelling | pubmed-99801412023-03-03 SEG: Segmentation Evaluation in absence of Ground truth labels Sims, Zachary Strgar, Luke Thirumalaisamy, Dharani Heussner, Robert Thibault, Guillaume Chang, Young Hwan bioRxiv Article Identifying individual cells or nuclei is often the first step in the analysis of multiplex tissue imaging (MTI) data. Recent efforts to produce plug-and-play, end-to-end MTI analysis tools such as MCMICRO(1)– though groundbreaking in their usability and extensibility – are often unable to provide users guidance regarding the most appropriate models for their segmentation task among an endless proliferation of novel segmentation methods. Unfortunately, evaluating segmentation results on a user’s dataset without ground truth labels is either purely subjective or eventually amounts to the task of performing the original, time-intensive annotation. As a consequence, researchers rely on models pre-trained on other large datasets for their unique tasks. Here, we propose a methodological approach for evaluating MTI nuclei segmentation methods in absence of ground truth labels by scoring relatively to a larger ensemble of segmentations. To avoid potential sensitivity to collective bias from the ensemble approach, we refine the ensemble via weighted average across segmentation methods, which we derive from a systematic model ablation study. First, we demonstrate a proof-of-concept and the feasibility of the proposed approach to evaluate segmentation performance in a small dataset with ground truth annotation. To validate the ensemble and demonstrate the importance of our method-specific weighting, we compare the ensemble’s detection and pixel-level predictions – derived without supervision - with the data’s ground truth labels. Second, we apply the methodology to an unlabeled larger tissue microarray (TMA) dataset, which includes a diverse set of breast cancer phenotypes, and provides decision guidelines for the general user to more easily choose the most suitable segmentation methods for their own dataset by systematically evaluating the performance of individual segmentation approaches in the entire dataset. Cold Spring Harbor Laboratory 2023-02-24 /pmc/articles/PMC9980141/ /pubmed/36865198 http://dx.doi.org/10.1101/2023.02.23.529809 Text en https://creativecommons.org/licenses/by-nc-nd/4.0/This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License (https://creativecommons.org/licenses/by-nc-nd/4.0/) , which allows reusers to copy and distribute the material in any medium or format in unadapted form only, for noncommercial purposes only, and only so long as attribution is given to the creator. |
spellingShingle | Article Sims, Zachary Strgar, Luke Thirumalaisamy, Dharani Heussner, Robert Thibault, Guillaume Chang, Young Hwan SEG: Segmentation Evaluation in absence of Ground truth labels |
title | SEG: Segmentation Evaluation in absence of Ground truth labels |
title_full | SEG: Segmentation Evaluation in absence of Ground truth labels |
title_fullStr | SEG: Segmentation Evaluation in absence of Ground truth labels |
title_full_unstemmed | SEG: Segmentation Evaluation in absence of Ground truth labels |
title_short | SEG: Segmentation Evaluation in absence of Ground truth labels |
title_sort | seg: segmentation evaluation in absence of ground truth labels |
topic | Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9980141/ https://www.ncbi.nlm.nih.gov/pubmed/36865198 http://dx.doi.org/10.1101/2023.02.23.529809 |
work_keys_str_mv | AT simszachary segsegmentationevaluationinabsenceofgroundtruthlabels AT strgarluke segsegmentationevaluationinabsenceofgroundtruthlabels AT thirumalaisamydharani segsegmentationevaluationinabsenceofgroundtruthlabels AT heussnerrobert segsegmentationevaluationinabsenceofgroundtruthlabels AT thibaultguillaume segsegmentationevaluationinabsenceofgroundtruthlabels AT changyounghwan segsegmentationevaluationinabsenceofgroundtruthlabels |