Cargando…

Using Background Knowledge from Preceding Studies for Building a Random Forest Prediction Model: A Plasmode Simulation Study

There is an increasing interest in machine learning (ML) algorithms for predicting patient outcomes, as these methods are designed to automatically discover complex data patterns. For example, the random forest (RF) algorithm is designed to identify relevant predictor variables out of a large set of...

Descripción completa

Detalles Bibliográficos
Autores principales: Hafermann, Lorena, Klein, Nadja, Rauch, Geraldine, Kammer, Michael, Heinze, Georg
Formato: Online Artículo Texto
Lenguaje:English
Publicado: MDPI 2022
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9222226/
https://www.ncbi.nlm.nih.gov/pubmed/35741566
http://dx.doi.org/10.3390/e24060847
_version_ 1784732822351642624
author Hafermann, Lorena
Klein, Nadja
Rauch, Geraldine
Kammer, Michael
Heinze, Georg
author_facet Hafermann, Lorena
Klein, Nadja
Rauch, Geraldine
Kammer, Michael
Heinze, Georg
author_sort Hafermann, Lorena
collection PubMed
description There is an increasing interest in machine learning (ML) algorithms for predicting patient outcomes, as these methods are designed to automatically discover complex data patterns. For example, the random forest (RF) algorithm is designed to identify relevant predictor variables out of a large set of candidates. In addition, researchers may also use external information for variable selection to improve model interpretability and variable selection accuracy, thereby prediction quality. However, it is unclear to which extent, if at all, RF and ML methods may benefit from external information. In this paper, we examine the usefulness of external information from prior variable selection studies that used traditional statistical modeling approaches such as the Lasso, or suboptimal methods such as univariate selection. We conducted a plasmode simulation study based on subsampling a data set from a pharmacoepidemiologic study with nearly 200,000 individuals, two binary outcomes and 1152 candidate predictor (mainly sparse binary) variables. When the scope of candidate predictors was reduced based on external knowledge RF models achieved better calibration, that is, better agreement of predictions and observed outcome rates. However, prediction quality measured by cross-entropy, AUROC or the Brier score did not improve. We recommend appraising the methodological quality of studies that serve as an external information source for future prediction model development.
format Online
Article
Text
id pubmed-9222226
institution National Center for Biotechnology Information
language English
publishDate 2022
publisher MDPI
record_format MEDLINE/PubMed
spelling pubmed-92222262022-06-24 Using Background Knowledge from Preceding Studies for Building a Random Forest Prediction Model: A Plasmode Simulation Study Hafermann, Lorena Klein, Nadja Rauch, Geraldine Kammer, Michael Heinze, Georg Entropy (Basel) Article There is an increasing interest in machine learning (ML) algorithms for predicting patient outcomes, as these methods are designed to automatically discover complex data patterns. For example, the random forest (RF) algorithm is designed to identify relevant predictor variables out of a large set of candidates. In addition, researchers may also use external information for variable selection to improve model interpretability and variable selection accuracy, thereby prediction quality. However, it is unclear to which extent, if at all, RF and ML methods may benefit from external information. In this paper, we examine the usefulness of external information from prior variable selection studies that used traditional statistical modeling approaches such as the Lasso, or suboptimal methods such as univariate selection. We conducted a plasmode simulation study based on subsampling a data set from a pharmacoepidemiologic study with nearly 200,000 individuals, two binary outcomes and 1152 candidate predictor (mainly sparse binary) variables. When the scope of candidate predictors was reduced based on external knowledge RF models achieved better calibration, that is, better agreement of predictions and observed outcome rates. However, prediction quality measured by cross-entropy, AUROC or the Brier score did not improve. We recommend appraising the methodological quality of studies that serve as an external information source for future prediction model development. MDPI 2022-06-20 /pmc/articles/PMC9222226/ /pubmed/35741566 http://dx.doi.org/10.3390/e24060847 Text en © 2022 by the authors. https://creativecommons.org/licenses/by/4.0/Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
spellingShingle Article
Hafermann, Lorena
Klein, Nadja
Rauch, Geraldine
Kammer, Michael
Heinze, Georg
Using Background Knowledge from Preceding Studies for Building a Random Forest Prediction Model: A Plasmode Simulation Study
title Using Background Knowledge from Preceding Studies for Building a Random Forest Prediction Model: A Plasmode Simulation Study
title_full Using Background Knowledge from Preceding Studies for Building a Random Forest Prediction Model: A Plasmode Simulation Study
title_fullStr Using Background Knowledge from Preceding Studies for Building a Random Forest Prediction Model: A Plasmode Simulation Study
title_full_unstemmed Using Background Knowledge from Preceding Studies for Building a Random Forest Prediction Model: A Plasmode Simulation Study
title_short Using Background Knowledge from Preceding Studies for Building a Random Forest Prediction Model: A Plasmode Simulation Study
title_sort using background knowledge from preceding studies for building a random forest prediction model: a plasmode simulation study
topic Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9222226/
https://www.ncbi.nlm.nih.gov/pubmed/35741566
http://dx.doi.org/10.3390/e24060847
work_keys_str_mv AT hafermannlorena usingbackgroundknowledgefromprecedingstudiesforbuildingarandomforestpredictionmodelaplasmodesimulationstudy
AT kleinnadja usingbackgroundknowledgefromprecedingstudiesforbuildingarandomforestpredictionmodelaplasmodesimulationstudy
AT rauchgeraldine usingbackgroundknowledgefromprecedingstudiesforbuildingarandomforestpredictionmodelaplasmodesimulationstudy
AT kammermichael usingbackgroundknowledgefromprecedingstudiesforbuildingarandomforestpredictionmodelaplasmodesimulationstudy
AT heinzegeorg usingbackgroundknowledgefromprecedingstudiesforbuildingarandomforestpredictionmodelaplasmodesimulationstudy