Cargando…
MedTric : A clinically applicable metric for evaluation of multi-label computational diagnostic systems
When judging the quality of a computational system for a pathological screening task, several factors seem to be important, like sensitivity, specificity, accuracy, etc. With machine learning based approaches showing promise in the multi-label paradigm, they are being widely adopted to diagnostics a...
Autores principales: | , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Public Library of Science
2023
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10414580/ https://www.ncbi.nlm.nih.gov/pubmed/37561695 http://dx.doi.org/10.1371/journal.pone.0283895 |
_version_ | 1785087369973596160 |
---|---|
author | Saha, Soumadeep Garain, Utpal Ukil, Arijit Pal, Arpan Khandelwal, Sundeep |
author_facet | Saha, Soumadeep Garain, Utpal Ukil, Arijit Pal, Arpan Khandelwal, Sundeep |
author_sort | Saha, Soumadeep |
collection | PubMed |
description | When judging the quality of a computational system for a pathological screening task, several factors seem to be important, like sensitivity, specificity, accuracy, etc. With machine learning based approaches showing promise in the multi-label paradigm, they are being widely adopted to diagnostics and digital therapeutics. Metrics are usually borrowed from machine learning literature, and the current consensus is to report results on a diverse set of metrics. It is infeasible to compare efficacy of computational systems which have been evaluated on different sets of metrics. From a diagnostic utility standpoint, the current metrics themselves are far from perfect, often biased by prevalence of negative samples or other statistical factors and importantly, they are designed to evaluate general purpose machine learning tasks. In this paper we outline the various parameters that are important in constructing a clinical metric aligned with diagnostic practice, and demonstrate their incompatibility with existing metrics. We propose a new metric, MedTric that takes into account several factors that are of clinical importance. MedTric is built from the ground up keeping in mind the unique context of computational diagnostics and the principle of risk minimization, penalizing missed diagnosis more harshly than over-diagnosis. MedTric is a unified metric for medical or pathological screening system evaluation. We compare this metric against other widely used metrics and demonstrate how our system outperforms them in key areas of medical relevance. |
format | Online Article Text |
id | pubmed-10414580 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2023 |
publisher | Public Library of Science |
record_format | MEDLINE/PubMed |
spelling | pubmed-104145802023-08-11 MedTric : A clinically applicable metric for evaluation of multi-label computational diagnostic systems Saha, Soumadeep Garain, Utpal Ukil, Arijit Pal, Arpan Khandelwal, Sundeep PLoS One Research Article When judging the quality of a computational system for a pathological screening task, several factors seem to be important, like sensitivity, specificity, accuracy, etc. With machine learning based approaches showing promise in the multi-label paradigm, they are being widely adopted to diagnostics and digital therapeutics. Metrics are usually borrowed from machine learning literature, and the current consensus is to report results on a diverse set of metrics. It is infeasible to compare efficacy of computational systems which have been evaluated on different sets of metrics. From a diagnostic utility standpoint, the current metrics themselves are far from perfect, often biased by prevalence of negative samples or other statistical factors and importantly, they are designed to evaluate general purpose machine learning tasks. In this paper we outline the various parameters that are important in constructing a clinical metric aligned with diagnostic practice, and demonstrate their incompatibility with existing metrics. We propose a new metric, MedTric that takes into account several factors that are of clinical importance. MedTric is built from the ground up keeping in mind the unique context of computational diagnostics and the principle of risk minimization, penalizing missed diagnosis more harshly than over-diagnosis. MedTric is a unified metric for medical or pathological screening system evaluation. We compare this metric against other widely used metrics and demonstrate how our system outperforms them in key areas of medical relevance. Public Library of Science 2023-08-10 /pmc/articles/PMC10414580/ /pubmed/37561695 http://dx.doi.org/10.1371/journal.pone.0283895 Text en © 2023 Saha et al https://creativecommons.org/licenses/by/4.0/This is an open access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/) , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. |
spellingShingle | Research Article Saha, Soumadeep Garain, Utpal Ukil, Arijit Pal, Arpan Khandelwal, Sundeep MedTric : A clinically applicable metric for evaluation of multi-label computational diagnostic systems |
title | MedTric : A clinically applicable metric for evaluation of multi-label computational diagnostic systems |
title_full | MedTric : A clinically applicable metric for evaluation of multi-label computational diagnostic systems |
title_fullStr | MedTric : A clinically applicable metric for evaluation of multi-label computational diagnostic systems |
title_full_unstemmed | MedTric : A clinically applicable metric for evaluation of multi-label computational diagnostic systems |
title_short | MedTric : A clinically applicable metric for evaluation of multi-label computational diagnostic systems |
title_sort | medtric : a clinically applicable metric for evaluation of multi-label computational diagnostic systems |
topic | Research Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10414580/ https://www.ncbi.nlm.nih.gov/pubmed/37561695 http://dx.doi.org/10.1371/journal.pone.0283895 |
work_keys_str_mv | AT sahasoumadeep medtricaclinicallyapplicablemetricforevaluationofmultilabelcomputationaldiagnosticsystems AT garainutpal medtricaclinicallyapplicablemetricforevaluationofmultilabelcomputationaldiagnosticsystems AT ukilarijit medtricaclinicallyapplicablemetricforevaluationofmultilabelcomputationaldiagnosticsystems AT palarpan medtricaclinicallyapplicablemetricforevaluationofmultilabelcomputationaldiagnosticsystems AT khandelwalsundeep medtricaclinicallyapplicablemetricforevaluationofmultilabelcomputationaldiagnosticsystems |