Cargando…

Framework for Testing Robustness of Machine Learning-Based Classifiers

There has been a rapid increase in the number of artificial intelligence (AI)/machine learning (ML)-based biomarker diagnostic classifiers in recent years. However, relatively little work has focused on assessing the robustness of these biomarkers, i.e., investigating the uncertainty of the AI/ML mo...

Descripción completa

Detalles Bibliográficos
Autores principales: Chuah, Joshua, Kruger, Uwe, Wang, Ge, Yan, Pingkun, Hahn, Juergen
Formato: Online Artículo Texto
Lenguaje:English
Publicado: MDPI 2022
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9409965/
https://www.ncbi.nlm.nih.gov/pubmed/36013263
http://dx.doi.org/10.3390/jpm12081314
_version_ 1784774979529736192
author Chuah, Joshua
Kruger, Uwe
Wang, Ge
Yan, Pingkun
Hahn, Juergen
author_facet Chuah, Joshua
Kruger, Uwe
Wang, Ge
Yan, Pingkun
Hahn, Juergen
author_sort Chuah, Joshua
collection PubMed
description There has been a rapid increase in the number of artificial intelligence (AI)/machine learning (ML)-based biomarker diagnostic classifiers in recent years. However, relatively little work has focused on assessing the robustness of these biomarkers, i.e., investigating the uncertainty of the AI/ML models that these biomarkers are based upon. This paper addresses this issue by proposing a framework to evaluate the already-developed classifiers with regard to their robustness by focusing on the variability of the classifiers’ performance and changes in the classifiers’ parameter values using factor analysis and Monte Carlo simulations. Specifically, this work evaluates (1) the importance of a classifier’s input features and (2) the variability of a classifier’s output and model parameter values in response to data perturbations. Additionally, it was found that one can estimate a priori how much replacement noise a classifier can tolerate while still meeting accuracy goals. To illustrate the evaluation framework, six different AI/ML-based biomarkers are developed using commonly used techniques (linear discriminant analysis, support vector machines, random forest, partial-least squares discriminant analysis, logistic regression, and multilayer perceptron) for a metabolomics dataset involving 24 measured metabolites taken from 159 study participants. The framework was able to correctly predict which of the classifiers should be less robust than others without recomputing the classifiers itself, and this prediction was then validated in a detailed analysis.
format Online
Article
Text
id pubmed-9409965
institution National Center for Biotechnology Information
language English
publishDate 2022
publisher MDPI
record_format MEDLINE/PubMed
spelling pubmed-94099652022-08-26 Framework for Testing Robustness of Machine Learning-Based Classifiers Chuah, Joshua Kruger, Uwe Wang, Ge Yan, Pingkun Hahn, Juergen J Pers Med Article There has been a rapid increase in the number of artificial intelligence (AI)/machine learning (ML)-based biomarker diagnostic classifiers in recent years. However, relatively little work has focused on assessing the robustness of these biomarkers, i.e., investigating the uncertainty of the AI/ML models that these biomarkers are based upon. This paper addresses this issue by proposing a framework to evaluate the already-developed classifiers with regard to their robustness by focusing on the variability of the classifiers’ performance and changes in the classifiers’ parameter values using factor analysis and Monte Carlo simulations. Specifically, this work evaluates (1) the importance of a classifier’s input features and (2) the variability of a classifier’s output and model parameter values in response to data perturbations. Additionally, it was found that one can estimate a priori how much replacement noise a classifier can tolerate while still meeting accuracy goals. To illustrate the evaluation framework, six different AI/ML-based biomarkers are developed using commonly used techniques (linear discriminant analysis, support vector machines, random forest, partial-least squares discriminant analysis, logistic regression, and multilayer perceptron) for a metabolomics dataset involving 24 measured metabolites taken from 159 study participants. The framework was able to correctly predict which of the classifiers should be less robust than others without recomputing the classifiers itself, and this prediction was then validated in a detailed analysis. MDPI 2022-08-14 /pmc/articles/PMC9409965/ /pubmed/36013263 http://dx.doi.org/10.3390/jpm12081314 Text en © 2022 by the authors. https://creativecommons.org/licenses/by/4.0/Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
spellingShingle Article
Chuah, Joshua
Kruger, Uwe
Wang, Ge
Yan, Pingkun
Hahn, Juergen
Framework for Testing Robustness of Machine Learning-Based Classifiers
title Framework for Testing Robustness of Machine Learning-Based Classifiers
title_full Framework for Testing Robustness of Machine Learning-Based Classifiers
title_fullStr Framework for Testing Robustness of Machine Learning-Based Classifiers
title_full_unstemmed Framework for Testing Robustness of Machine Learning-Based Classifiers
title_short Framework for Testing Robustness of Machine Learning-Based Classifiers
title_sort framework for testing robustness of machine learning-based classifiers
topic Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9409965/
https://www.ncbi.nlm.nih.gov/pubmed/36013263
http://dx.doi.org/10.3390/jpm12081314
work_keys_str_mv AT chuahjoshua frameworkfortestingrobustnessofmachinelearningbasedclassifiers
AT krugeruwe frameworkfortestingrobustnessofmachinelearningbasedclassifiers
AT wangge frameworkfortestingrobustnessofmachinelearningbasedclassifiers
AT yanpingkun frameworkfortestingrobustnessofmachinelearningbasedclassifiers
AT hahnjuergen frameworkfortestingrobustnessofmachinelearningbasedclassifiers