Cargando…
Using Background Knowledge from Preceding Studies for Building a Random Forest Prediction Model: A Plasmode Simulation Study
There is an increasing interest in machine learning (ML) algorithms for predicting patient outcomes, as these methods are designed to automatically discover complex data patterns. For example, the random forest (RF) algorithm is designed to identify relevant predictor variables out of a large set of...
Autores principales: | , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
MDPI
2022
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9222226/ https://www.ncbi.nlm.nih.gov/pubmed/35741566 http://dx.doi.org/10.3390/e24060847 |
_version_ | 1784732822351642624 |
---|---|
author | Hafermann, Lorena Klein, Nadja Rauch, Geraldine Kammer, Michael Heinze, Georg |
author_facet | Hafermann, Lorena Klein, Nadja Rauch, Geraldine Kammer, Michael Heinze, Georg |
author_sort | Hafermann, Lorena |
collection | PubMed |
description | There is an increasing interest in machine learning (ML) algorithms for predicting patient outcomes, as these methods are designed to automatically discover complex data patterns. For example, the random forest (RF) algorithm is designed to identify relevant predictor variables out of a large set of candidates. In addition, researchers may also use external information for variable selection to improve model interpretability and variable selection accuracy, thereby prediction quality. However, it is unclear to which extent, if at all, RF and ML methods may benefit from external information. In this paper, we examine the usefulness of external information from prior variable selection studies that used traditional statistical modeling approaches such as the Lasso, or suboptimal methods such as univariate selection. We conducted a plasmode simulation study based on subsampling a data set from a pharmacoepidemiologic study with nearly 200,000 individuals, two binary outcomes and 1152 candidate predictor (mainly sparse binary) variables. When the scope of candidate predictors was reduced based on external knowledge RF models achieved better calibration, that is, better agreement of predictions and observed outcome rates. However, prediction quality measured by cross-entropy, AUROC or the Brier score did not improve. We recommend appraising the methodological quality of studies that serve as an external information source for future prediction model development. |
format | Online Article Text |
id | pubmed-9222226 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2022 |
publisher | MDPI |
record_format | MEDLINE/PubMed |
spelling | pubmed-92222262022-06-24 Using Background Knowledge from Preceding Studies for Building a Random Forest Prediction Model: A Plasmode Simulation Study Hafermann, Lorena Klein, Nadja Rauch, Geraldine Kammer, Michael Heinze, Georg Entropy (Basel) Article There is an increasing interest in machine learning (ML) algorithms for predicting patient outcomes, as these methods are designed to automatically discover complex data patterns. For example, the random forest (RF) algorithm is designed to identify relevant predictor variables out of a large set of candidates. In addition, researchers may also use external information for variable selection to improve model interpretability and variable selection accuracy, thereby prediction quality. However, it is unclear to which extent, if at all, RF and ML methods may benefit from external information. In this paper, we examine the usefulness of external information from prior variable selection studies that used traditional statistical modeling approaches such as the Lasso, or suboptimal methods such as univariate selection. We conducted a plasmode simulation study based on subsampling a data set from a pharmacoepidemiologic study with nearly 200,000 individuals, two binary outcomes and 1152 candidate predictor (mainly sparse binary) variables. When the scope of candidate predictors was reduced based on external knowledge RF models achieved better calibration, that is, better agreement of predictions and observed outcome rates. However, prediction quality measured by cross-entropy, AUROC or the Brier score did not improve. We recommend appraising the methodological quality of studies that serve as an external information source for future prediction model development. MDPI 2022-06-20 /pmc/articles/PMC9222226/ /pubmed/35741566 http://dx.doi.org/10.3390/e24060847 Text en © 2022 by the authors. https://creativecommons.org/licenses/by/4.0/Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/). |
spellingShingle | Article Hafermann, Lorena Klein, Nadja Rauch, Geraldine Kammer, Michael Heinze, Georg Using Background Knowledge from Preceding Studies for Building a Random Forest Prediction Model: A Plasmode Simulation Study |
title | Using Background Knowledge from Preceding Studies for Building a Random Forest Prediction Model: A Plasmode Simulation Study |
title_full | Using Background Knowledge from Preceding Studies for Building a Random Forest Prediction Model: A Plasmode Simulation Study |
title_fullStr | Using Background Knowledge from Preceding Studies for Building a Random Forest Prediction Model: A Plasmode Simulation Study |
title_full_unstemmed | Using Background Knowledge from Preceding Studies for Building a Random Forest Prediction Model: A Plasmode Simulation Study |
title_short | Using Background Knowledge from Preceding Studies for Building a Random Forest Prediction Model: A Plasmode Simulation Study |
title_sort | using background knowledge from preceding studies for building a random forest prediction model: a plasmode simulation study |
topic | Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9222226/ https://www.ncbi.nlm.nih.gov/pubmed/35741566 http://dx.doi.org/10.3390/e24060847 |
work_keys_str_mv | AT hafermannlorena usingbackgroundknowledgefromprecedingstudiesforbuildingarandomforestpredictionmodelaplasmodesimulationstudy AT kleinnadja usingbackgroundknowledgefromprecedingstudiesforbuildingarandomforestpredictionmodelaplasmodesimulationstudy AT rauchgeraldine usingbackgroundknowledgefromprecedingstudiesforbuildingarandomforestpredictionmodelaplasmodesimulationstudy AT kammermichael usingbackgroundknowledgefromprecedingstudiesforbuildingarandomforestpredictionmodelaplasmodesimulationstudy AT heinzegeorg usingbackgroundknowledgefromprecedingstudiesforbuildingarandomforestpredictionmodelaplasmodesimulationstudy |