Cargando…

Challenges in the real world use of classification accuracy metrics: From recall and precision to the Matthews correlation coefficient

The accuracy of a classification is fundamental to its interpretation, use and ultimately decision making. Unfortunately, the apparent accuracy assessed can differ greatly from the true accuracy. Mis-estimation of classification accuracy metrics and associated mis-interpretations are often due to va...

Descripción completa

Detalles Bibliográficos
Autor principal: Foody, Giles M.
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Public Library of Science 2023
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10550141/
https://www.ncbi.nlm.nih.gov/pubmed/37792898
http://dx.doi.org/10.1371/journal.pone.0291908
_version_ 1785115471032352768
author Foody, Giles M.
author_facet Foody, Giles M.
author_sort Foody, Giles M.
collection PubMed
description The accuracy of a classification is fundamental to its interpretation, use and ultimately decision making. Unfortunately, the apparent accuracy assessed can differ greatly from the true accuracy. Mis-estimation of classification accuracy metrics and associated mis-interpretations are often due to variations in prevalence and the use of an imperfect reference standard. The fundamental issues underlying the problems associated with variations in prevalence and reference standard quality are revisited here for binary classifications with particular attention focused on the use of the Matthews correlation coefficient (MCC). A key attribute claimed of the MCC is that a high value can only be attained when the classification performed well on both classes in a binary classification. However, it is shown here that the apparent magnitude of a set of popular accuracy metrics used in fields such as computer science medicine and environmental science (Recall, Precision, Specificity, Negative Predictive Value, J, F(1), likelihood ratios and MCC) and one key attribute (prevalence) were all influenced greatly by variations in prevalence and use of an imperfect reference standard. Simulations using realistic values for data quality in applications such as remote sensing showed each metric varied over the range of possible prevalence and at differing levels of reference standard quality. The direction and magnitude of accuracy metric mis-estimation were a function of prevalence and the size and nature of the imperfections in the reference standard. It was evident that the apparent MCC could be substantially under- or over-estimated. Additionally, a high apparent MCC arose from an unquestionably poor classification. As with some other metrics of accuracy, the utility of the MCC may be overstated and apparent values need to be interpreted with caution. Apparent accuracy and prevalence values can be mis-leading and calls for the issues to be recognised and addressed should be heeded.
format Online
Article
Text
id pubmed-10550141
institution National Center for Biotechnology Information
language English
publishDate 2023
publisher Public Library of Science
record_format MEDLINE/PubMed
spelling pubmed-105501412023-10-05 Challenges in the real world use of classification accuracy metrics: From recall and precision to the Matthews correlation coefficient Foody, Giles M. PLoS One Research Article The accuracy of a classification is fundamental to its interpretation, use and ultimately decision making. Unfortunately, the apparent accuracy assessed can differ greatly from the true accuracy. Mis-estimation of classification accuracy metrics and associated mis-interpretations are often due to variations in prevalence and the use of an imperfect reference standard. The fundamental issues underlying the problems associated with variations in prevalence and reference standard quality are revisited here for binary classifications with particular attention focused on the use of the Matthews correlation coefficient (MCC). A key attribute claimed of the MCC is that a high value can only be attained when the classification performed well on both classes in a binary classification. However, it is shown here that the apparent magnitude of a set of popular accuracy metrics used in fields such as computer science medicine and environmental science (Recall, Precision, Specificity, Negative Predictive Value, J, F(1), likelihood ratios and MCC) and one key attribute (prevalence) were all influenced greatly by variations in prevalence and use of an imperfect reference standard. Simulations using realistic values for data quality in applications such as remote sensing showed each metric varied over the range of possible prevalence and at differing levels of reference standard quality. The direction and magnitude of accuracy metric mis-estimation were a function of prevalence and the size and nature of the imperfections in the reference standard. It was evident that the apparent MCC could be substantially under- or over-estimated. Additionally, a high apparent MCC arose from an unquestionably poor classification. As with some other metrics of accuracy, the utility of the MCC may be overstated and apparent values need to be interpreted with caution. Apparent accuracy and prevalence values can be mis-leading and calls for the issues to be recognised and addressed should be heeded. Public Library of Science 2023-10-04 /pmc/articles/PMC10550141/ /pubmed/37792898 http://dx.doi.org/10.1371/journal.pone.0291908 Text en © 2023 Giles M. Foody https://creativecommons.org/licenses/by/4.0/This is an open access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/) , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
spellingShingle Research Article
Foody, Giles M.
Challenges in the real world use of classification accuracy metrics: From recall and precision to the Matthews correlation coefficient
title Challenges in the real world use of classification accuracy metrics: From recall and precision to the Matthews correlation coefficient
title_full Challenges in the real world use of classification accuracy metrics: From recall and precision to the Matthews correlation coefficient
title_fullStr Challenges in the real world use of classification accuracy metrics: From recall and precision to the Matthews correlation coefficient
title_full_unstemmed Challenges in the real world use of classification accuracy metrics: From recall and precision to the Matthews correlation coefficient
title_short Challenges in the real world use of classification accuracy metrics: From recall and precision to the Matthews correlation coefficient
title_sort challenges in the real world use of classification accuracy metrics: from recall and precision to the matthews correlation coefficient
topic Research Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10550141/
https://www.ncbi.nlm.nih.gov/pubmed/37792898
http://dx.doi.org/10.1371/journal.pone.0291908
work_keys_str_mv AT foodygilesm challengesintherealworlduseofclassificationaccuracymetricsfromrecallandprecisiontothematthewscorrelationcoefficient