Cargando…
Framework for Testing Robustness of Machine Learning-Based Classifiers
There has been a rapid increase in the number of artificial intelligence (AI)/machine learning (ML)-based biomarker diagnostic classifiers in recent years. However, relatively little work has focused on assessing the robustness of these biomarkers, i.e., investigating the uncertainty of the AI/ML mo...
Autores principales: | , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
MDPI
2022
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9409965/ https://www.ncbi.nlm.nih.gov/pubmed/36013263 http://dx.doi.org/10.3390/jpm12081314 |
_version_ | 1784774979529736192 |
---|---|
author | Chuah, Joshua Kruger, Uwe Wang, Ge Yan, Pingkun Hahn, Juergen |
author_facet | Chuah, Joshua Kruger, Uwe Wang, Ge Yan, Pingkun Hahn, Juergen |
author_sort | Chuah, Joshua |
collection | PubMed |
description | There has been a rapid increase in the number of artificial intelligence (AI)/machine learning (ML)-based biomarker diagnostic classifiers in recent years. However, relatively little work has focused on assessing the robustness of these biomarkers, i.e., investigating the uncertainty of the AI/ML models that these biomarkers are based upon. This paper addresses this issue by proposing a framework to evaluate the already-developed classifiers with regard to their robustness by focusing on the variability of the classifiers’ performance and changes in the classifiers’ parameter values using factor analysis and Monte Carlo simulations. Specifically, this work evaluates (1) the importance of a classifier’s input features and (2) the variability of a classifier’s output and model parameter values in response to data perturbations. Additionally, it was found that one can estimate a priori how much replacement noise a classifier can tolerate while still meeting accuracy goals. To illustrate the evaluation framework, six different AI/ML-based biomarkers are developed using commonly used techniques (linear discriminant analysis, support vector machines, random forest, partial-least squares discriminant analysis, logistic regression, and multilayer perceptron) for a metabolomics dataset involving 24 measured metabolites taken from 159 study participants. The framework was able to correctly predict which of the classifiers should be less robust than others without recomputing the classifiers itself, and this prediction was then validated in a detailed analysis. |
format | Online Article Text |
id | pubmed-9409965 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2022 |
publisher | MDPI |
record_format | MEDLINE/PubMed |
spelling | pubmed-94099652022-08-26 Framework for Testing Robustness of Machine Learning-Based Classifiers Chuah, Joshua Kruger, Uwe Wang, Ge Yan, Pingkun Hahn, Juergen J Pers Med Article There has been a rapid increase in the number of artificial intelligence (AI)/machine learning (ML)-based biomarker diagnostic classifiers in recent years. However, relatively little work has focused on assessing the robustness of these biomarkers, i.e., investigating the uncertainty of the AI/ML models that these biomarkers are based upon. This paper addresses this issue by proposing a framework to evaluate the already-developed classifiers with regard to their robustness by focusing on the variability of the classifiers’ performance and changes in the classifiers’ parameter values using factor analysis and Monte Carlo simulations. Specifically, this work evaluates (1) the importance of a classifier’s input features and (2) the variability of a classifier’s output and model parameter values in response to data perturbations. Additionally, it was found that one can estimate a priori how much replacement noise a classifier can tolerate while still meeting accuracy goals. To illustrate the evaluation framework, six different AI/ML-based biomarkers are developed using commonly used techniques (linear discriminant analysis, support vector machines, random forest, partial-least squares discriminant analysis, logistic regression, and multilayer perceptron) for a metabolomics dataset involving 24 measured metabolites taken from 159 study participants. The framework was able to correctly predict which of the classifiers should be less robust than others without recomputing the classifiers itself, and this prediction was then validated in a detailed analysis. MDPI 2022-08-14 /pmc/articles/PMC9409965/ /pubmed/36013263 http://dx.doi.org/10.3390/jpm12081314 Text en © 2022 by the authors. https://creativecommons.org/licenses/by/4.0/Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/). |
spellingShingle | Article Chuah, Joshua Kruger, Uwe Wang, Ge Yan, Pingkun Hahn, Juergen Framework for Testing Robustness of Machine Learning-Based Classifiers |
title | Framework for Testing Robustness of Machine Learning-Based Classifiers |
title_full | Framework for Testing Robustness of Machine Learning-Based Classifiers |
title_fullStr | Framework for Testing Robustness of Machine Learning-Based Classifiers |
title_full_unstemmed | Framework for Testing Robustness of Machine Learning-Based Classifiers |
title_short | Framework for Testing Robustness of Machine Learning-Based Classifiers |
title_sort | framework for testing robustness of machine learning-based classifiers |
topic | Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9409965/ https://www.ncbi.nlm.nih.gov/pubmed/36013263 http://dx.doi.org/10.3390/jpm12081314 |
work_keys_str_mv | AT chuahjoshua frameworkfortestingrobustnessofmachinelearningbasedclassifiers AT krugeruwe frameworkfortestingrobustnessofmachinelearningbasedclassifiers AT wangge frameworkfortestingrobustnessofmachinelearningbasedclassifiers AT yanpingkun frameworkfortestingrobustnessofmachinelearningbasedclassifiers AT hahnjuergen frameworkfortestingrobustnessofmachinelearningbasedclassifiers |