Cargando…
Impact of the Choice of Cross-Validation Techniques on the Results of Machine Learning-Based Diagnostic Applications
OBJECTIVES: With advances in data availability and computing capabilities, artificial intelligence and machine learning technologies have evolved rapidly in recent years. Researchers have taken advantage of these developments in healthcare informatics and created reliable tools to predict or classif...
Autores principales: | , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Korean Society of Medical Informatics
2021
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8369053/ https://www.ncbi.nlm.nih.gov/pubmed/34384201 http://dx.doi.org/10.4258/hir.2021.27.3.189 |
_version_ | 1783739208794374144 |
---|---|
author | Tougui, Ilias Jilbab, Abdelilah El Mhamdi, Jamal |
author_facet | Tougui, Ilias Jilbab, Abdelilah El Mhamdi, Jamal |
author_sort | Tougui, Ilias |
collection | PubMed |
description | OBJECTIVES: With advances in data availability and computing capabilities, artificial intelligence and machine learning technologies have evolved rapidly in recent years. Researchers have taken advantage of these developments in healthcare informatics and created reliable tools to predict or classify diseases using machine learning-based algorithms. To correctly quantify the performance of those algorithms, the standard approach is to use cross-validation, where the algorithm is trained on a training set, and its performance is measured on a validation set. Both datasets should be subject-independent to simulate the expected behavior of a clinical study. This study compares two cross-validation strategies, the subject-wise and the record-wise techniques; the subject-wise strategy correctly mimics the process of a clinical study, while the record-wise strategy does not. METHODS: We started by creating a dataset of smartphone audio recordings of subjects diagnosed with and without Parkinson’s disease. This dataset was then divided into training and holdout sets using subject-wise and the record-wise divisions. The training set was used to measure the performance of two classifiers (support vector machine and random forest) to compare six cross-validation techniques that simulated either the subject-wise process or the record-wise process. The holdout set was used to calculate the true error of the classifiers. RESULTS: The record-wise division and the record-wise cross-validation techniques overestimated the performance of the classifiers and underestimated the classification error. CONCLUSIONS: In a diagnostic scenario, the subject-wise technique is the proper way of estimating a model’s performance, and record-wise techniques should be avoided. |
format | Online Article Text |
id | pubmed-8369053 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2021 |
publisher | Korean Society of Medical Informatics |
record_format | MEDLINE/PubMed |
spelling | pubmed-83690532021-08-26 Impact of the Choice of Cross-Validation Techniques on the Results of Machine Learning-Based Diagnostic Applications Tougui, Ilias Jilbab, Abdelilah El Mhamdi, Jamal Healthc Inform Res Original Article OBJECTIVES: With advances in data availability and computing capabilities, artificial intelligence and machine learning technologies have evolved rapidly in recent years. Researchers have taken advantage of these developments in healthcare informatics and created reliable tools to predict or classify diseases using machine learning-based algorithms. To correctly quantify the performance of those algorithms, the standard approach is to use cross-validation, where the algorithm is trained on a training set, and its performance is measured on a validation set. Both datasets should be subject-independent to simulate the expected behavior of a clinical study. This study compares two cross-validation strategies, the subject-wise and the record-wise techniques; the subject-wise strategy correctly mimics the process of a clinical study, while the record-wise strategy does not. METHODS: We started by creating a dataset of smartphone audio recordings of subjects diagnosed with and without Parkinson’s disease. This dataset was then divided into training and holdout sets using subject-wise and the record-wise divisions. The training set was used to measure the performance of two classifiers (support vector machine and random forest) to compare six cross-validation techniques that simulated either the subject-wise process or the record-wise process. The holdout set was used to calculate the true error of the classifiers. RESULTS: The record-wise division and the record-wise cross-validation techniques overestimated the performance of the classifiers and underestimated the classification error. CONCLUSIONS: In a diagnostic scenario, the subject-wise technique is the proper way of estimating a model’s performance, and record-wise techniques should be avoided. Korean Society of Medical Informatics 2021-07 2021-07-31 /pmc/articles/PMC8369053/ /pubmed/34384201 http://dx.doi.org/10.4258/hir.2021.27.3.189 Text en © 2021 The Korean Society of Medical Informatics https://creativecommons.org/licenses/by-nc/4.0/This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0/ (https://creativecommons.org/licenses/by-nc/4.0/) ) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited. |
spellingShingle | Original Article Tougui, Ilias Jilbab, Abdelilah El Mhamdi, Jamal Impact of the Choice of Cross-Validation Techniques on the Results of Machine Learning-Based Diagnostic Applications |
title | Impact of the Choice of Cross-Validation Techniques on the Results of Machine Learning-Based Diagnostic Applications |
title_full | Impact of the Choice of Cross-Validation Techniques on the Results of Machine Learning-Based Diagnostic Applications |
title_fullStr | Impact of the Choice of Cross-Validation Techniques on the Results of Machine Learning-Based Diagnostic Applications |
title_full_unstemmed | Impact of the Choice of Cross-Validation Techniques on the Results of Machine Learning-Based Diagnostic Applications |
title_short | Impact of the Choice of Cross-Validation Techniques on the Results of Machine Learning-Based Diagnostic Applications |
title_sort | impact of the choice of cross-validation techniques on the results of machine learning-based diagnostic applications |
topic | Original Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8369053/ https://www.ncbi.nlm.nih.gov/pubmed/34384201 http://dx.doi.org/10.4258/hir.2021.27.3.189 |
work_keys_str_mv | AT touguiilias impactofthechoiceofcrossvalidationtechniquesontheresultsofmachinelearningbaseddiagnosticapplications AT jilbababdelilah impactofthechoiceofcrossvalidationtechniquesontheresultsofmachinelearningbaseddiagnosticapplications AT elmhamdijamal impactofthechoiceofcrossvalidationtechniquesontheresultsofmachinelearningbaseddiagnosticapplications |