Cargando…

SEG: Segmentation Evaluation in absence of Ground truth labels

Identifying individual cells or nuclei is often the first step in the analysis of multiplex tissue imaging (MTI) data. Recent efforts to produce plug-and-play, end-to-end MTI analysis tools such as MCMICRO(1)– though groundbreaking in their usability and extensibility – are often unable to provide u...

Descripción completa

Detalles Bibliográficos
Autores principales: Sims, Zachary, Strgar, Luke, Thirumalaisamy, Dharani, Heussner, Robert, Thibault, Guillaume, Chang, Young Hwan
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Cold Spring Harbor Laboratory 2023
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9980141/
https://www.ncbi.nlm.nih.gov/pubmed/36865198
http://dx.doi.org/10.1101/2023.02.23.529809
_version_ 1784899856263806976
author Sims, Zachary
Strgar, Luke
Thirumalaisamy, Dharani
Heussner, Robert
Thibault, Guillaume
Chang, Young Hwan
author_facet Sims, Zachary
Strgar, Luke
Thirumalaisamy, Dharani
Heussner, Robert
Thibault, Guillaume
Chang, Young Hwan
author_sort Sims, Zachary
collection PubMed
description Identifying individual cells or nuclei is often the first step in the analysis of multiplex tissue imaging (MTI) data. Recent efforts to produce plug-and-play, end-to-end MTI analysis tools such as MCMICRO(1)– though groundbreaking in their usability and extensibility – are often unable to provide users guidance regarding the most appropriate models for their segmentation task among an endless proliferation of novel segmentation methods. Unfortunately, evaluating segmentation results on a user’s dataset without ground truth labels is either purely subjective or eventually amounts to the task of performing the original, time-intensive annotation. As a consequence, researchers rely on models pre-trained on other large datasets for their unique tasks. Here, we propose a methodological approach for evaluating MTI nuclei segmentation methods in absence of ground truth labels by scoring relatively to a larger ensemble of segmentations. To avoid potential sensitivity to collective bias from the ensemble approach, we refine the ensemble via weighted average across segmentation methods, which we derive from a systematic model ablation study. First, we demonstrate a proof-of-concept and the feasibility of the proposed approach to evaluate segmentation performance in a small dataset with ground truth annotation. To validate the ensemble and demonstrate the importance of our method-specific weighting, we compare the ensemble’s detection and pixel-level predictions – derived without supervision - with the data’s ground truth labels. Second, we apply the methodology to an unlabeled larger tissue microarray (TMA) dataset, which includes a diverse set of breast cancer phenotypes, and provides decision guidelines for the general user to more easily choose the most suitable segmentation methods for their own dataset by systematically evaluating the performance of individual segmentation approaches in the entire dataset.
format Online
Article
Text
id pubmed-9980141
institution National Center for Biotechnology Information
language English
publishDate 2023
publisher Cold Spring Harbor Laboratory
record_format MEDLINE/PubMed
spelling pubmed-99801412023-03-03 SEG: Segmentation Evaluation in absence of Ground truth labels Sims, Zachary Strgar, Luke Thirumalaisamy, Dharani Heussner, Robert Thibault, Guillaume Chang, Young Hwan bioRxiv Article Identifying individual cells or nuclei is often the first step in the analysis of multiplex tissue imaging (MTI) data. Recent efforts to produce plug-and-play, end-to-end MTI analysis tools such as MCMICRO(1)– though groundbreaking in their usability and extensibility – are often unable to provide users guidance regarding the most appropriate models for their segmentation task among an endless proliferation of novel segmentation methods. Unfortunately, evaluating segmentation results on a user’s dataset without ground truth labels is either purely subjective or eventually amounts to the task of performing the original, time-intensive annotation. As a consequence, researchers rely on models pre-trained on other large datasets for their unique tasks. Here, we propose a methodological approach for evaluating MTI nuclei segmentation methods in absence of ground truth labels by scoring relatively to a larger ensemble of segmentations. To avoid potential sensitivity to collective bias from the ensemble approach, we refine the ensemble via weighted average across segmentation methods, which we derive from a systematic model ablation study. First, we demonstrate a proof-of-concept and the feasibility of the proposed approach to evaluate segmentation performance in a small dataset with ground truth annotation. To validate the ensemble and demonstrate the importance of our method-specific weighting, we compare the ensemble’s detection and pixel-level predictions – derived without supervision - with the data’s ground truth labels. Second, we apply the methodology to an unlabeled larger tissue microarray (TMA) dataset, which includes a diverse set of breast cancer phenotypes, and provides decision guidelines for the general user to more easily choose the most suitable segmentation methods for their own dataset by systematically evaluating the performance of individual segmentation approaches in the entire dataset. Cold Spring Harbor Laboratory 2023-02-24 /pmc/articles/PMC9980141/ /pubmed/36865198 http://dx.doi.org/10.1101/2023.02.23.529809 Text en https://creativecommons.org/licenses/by-nc-nd/4.0/This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License (https://creativecommons.org/licenses/by-nc-nd/4.0/) , which allows reusers to copy and distribute the material in any medium or format in unadapted form only, for noncommercial purposes only, and only so long as attribution is given to the creator.
spellingShingle Article
Sims, Zachary
Strgar, Luke
Thirumalaisamy, Dharani
Heussner, Robert
Thibault, Guillaume
Chang, Young Hwan
SEG: Segmentation Evaluation in absence of Ground truth labels
title SEG: Segmentation Evaluation in absence of Ground truth labels
title_full SEG: Segmentation Evaluation in absence of Ground truth labels
title_fullStr SEG: Segmentation Evaluation in absence of Ground truth labels
title_full_unstemmed SEG: Segmentation Evaluation in absence of Ground truth labels
title_short SEG: Segmentation Evaluation in absence of Ground truth labels
title_sort seg: segmentation evaluation in absence of ground truth labels
topic Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9980141/
https://www.ncbi.nlm.nih.gov/pubmed/36865198
http://dx.doi.org/10.1101/2023.02.23.529809
work_keys_str_mv AT simszachary segsegmentationevaluationinabsenceofgroundtruthlabels
AT strgarluke segsegmentationevaluationinabsenceofgroundtruthlabels
AT thirumalaisamydharani segsegmentationevaluationinabsenceofgroundtruthlabels
AT heussnerrobert segsegmentationevaluationinabsenceofgroundtruthlabels
AT thibaultguillaume segsegmentationevaluationinabsenceofgroundtruthlabels
AT changyounghwan segsegmentationevaluationinabsenceofgroundtruthlabels