Cargando…

Design and implementation of a standardized framework to generate and evaluate patient-level prediction models using observational healthcare data

OBJECTIVE: To develop a conceptual prediction model framework containing standardized steps and describe the corresponding open-source software developed to consistently implement the framework across computational environments and observational healthcare databases to enable model sharing and repro...

Descripción completa

Detalles Bibliográficos
Autores principales: Reps, Jenna M, Schuemie, Martijn J, Suchard, Marc A, Ryan, Patrick B, Rijnbeek, Peter R
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Oxford University Press 2018
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6077830/
https://www.ncbi.nlm.nih.gov/pubmed/29718407
http://dx.doi.org/10.1093/jamia/ocy032
_version_ 1783344988527001600
author Reps, Jenna M
Schuemie, Martijn J
Suchard, Marc A
Ryan, Patrick B
Rijnbeek, Peter R
author_facet Reps, Jenna M
Schuemie, Martijn J
Suchard, Marc A
Ryan, Patrick B
Rijnbeek, Peter R
author_sort Reps, Jenna M
collection PubMed
description OBJECTIVE: To develop a conceptual prediction model framework containing standardized steps and describe the corresponding open-source software developed to consistently implement the framework across computational environments and observational healthcare databases to enable model sharing and reproducibility. METHODS: Based on existing best practices we propose a 5 step standardized framework for: (1) transparently defining the problem; (2) selecting suitable datasets; (3) constructing variables from the observational data; (4) learning the predictive model; and (5) validating the model performance. We implemented this framework as open-source software utilizing the Observational Medical Outcomes Partnership Common Data Model to enable convenient sharing of models and reproduction of model evaluation across multiple observational datasets. The software implementation contains default covariates and classifiers but the framework enables customization and extension. RESULTS: As a proof-of-concept, demonstrating the transparency and ease of model dissemination using the software, we developed prediction models for 21 different outcomes within a target population of people suffering from depression across 4 observational databases. All 84 models are available in an accessible online repository to be implemented by anyone with access to an observational database in the Common Data Model format. CONCLUSIONS: The proof-of-concept study illustrates the framework’s ability to develop reproducible models that can be readily shared and offers the potential to perform extensive external validation of models, and improve their likelihood of clinical uptake. In future work the framework will be applied to perform an “all-by-all” prediction analysis to assess the observational data prediction domain across numerous target populations, outcomes and time, and risk settings.
format Online
Article
Text
id pubmed-6077830
institution National Center for Biotechnology Information
language English
publishDate 2018
publisher Oxford University Press
record_format MEDLINE/PubMed
spelling pubmed-60778302018-08-09 Design and implementation of a standardized framework to generate and evaluate patient-level prediction models using observational healthcare data Reps, Jenna M Schuemie, Martijn J Suchard, Marc A Ryan, Patrick B Rijnbeek, Peter R J Am Med Inform Assoc Research and Applications OBJECTIVE: To develop a conceptual prediction model framework containing standardized steps and describe the corresponding open-source software developed to consistently implement the framework across computational environments and observational healthcare databases to enable model sharing and reproducibility. METHODS: Based on existing best practices we propose a 5 step standardized framework for: (1) transparently defining the problem; (2) selecting suitable datasets; (3) constructing variables from the observational data; (4) learning the predictive model; and (5) validating the model performance. We implemented this framework as open-source software utilizing the Observational Medical Outcomes Partnership Common Data Model to enable convenient sharing of models and reproduction of model evaluation across multiple observational datasets. The software implementation contains default covariates and classifiers but the framework enables customization and extension. RESULTS: As a proof-of-concept, demonstrating the transparency and ease of model dissemination using the software, we developed prediction models for 21 different outcomes within a target population of people suffering from depression across 4 observational databases. All 84 models are available in an accessible online repository to be implemented by anyone with access to an observational database in the Common Data Model format. CONCLUSIONS: The proof-of-concept study illustrates the framework’s ability to develop reproducible models that can be readily shared and offers the potential to perform extensive external validation of models, and improve their likelihood of clinical uptake. In future work the framework will be applied to perform an “all-by-all” prediction analysis to assess the observational data prediction domain across numerous target populations, outcomes and time, and risk settings. Oxford University Press 2018-04-27 /pmc/articles/PMC6077830/ /pubmed/29718407 http://dx.doi.org/10.1093/jamia/ocy032 Text en © The Author(s) 2018. Published by Oxford University Press on behalf of the American Medical Informatics Association. http://creativecommons.org/licenses/by-nc/4.0/ This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited. For commercial re-use, please contact journals.permissions@oup.com
spellingShingle Research and Applications
Reps, Jenna M
Schuemie, Martijn J
Suchard, Marc A
Ryan, Patrick B
Rijnbeek, Peter R
Design and implementation of a standardized framework to generate and evaluate patient-level prediction models using observational healthcare data
title Design and implementation of a standardized framework to generate and evaluate patient-level prediction models using observational healthcare data
title_full Design and implementation of a standardized framework to generate and evaluate patient-level prediction models using observational healthcare data
title_fullStr Design and implementation of a standardized framework to generate and evaluate patient-level prediction models using observational healthcare data
title_full_unstemmed Design and implementation of a standardized framework to generate and evaluate patient-level prediction models using observational healthcare data
title_short Design and implementation of a standardized framework to generate and evaluate patient-level prediction models using observational healthcare data
title_sort design and implementation of a standardized framework to generate and evaluate patient-level prediction models using observational healthcare data
topic Research and Applications
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6077830/
https://www.ncbi.nlm.nih.gov/pubmed/29718407
http://dx.doi.org/10.1093/jamia/ocy032
work_keys_str_mv AT repsjennam designandimplementationofastandardizedframeworktogenerateandevaluatepatientlevelpredictionmodelsusingobservationalhealthcaredata
AT schuemiemartijnj designandimplementationofastandardizedframeworktogenerateandevaluatepatientlevelpredictionmodelsusingobservationalhealthcaredata
AT suchardmarca designandimplementationofastandardizedframeworktogenerateandevaluatepatientlevelpredictionmodelsusingobservationalhealthcaredata
AT ryanpatrickb designandimplementationofastandardizedframeworktogenerateandevaluatepatientlevelpredictionmodelsusingobservationalhealthcaredata
AT rijnbeekpeterr designandimplementationofastandardizedframeworktogenerateandevaluatepatientlevelpredictionmodelsusingobservationalhealthcaredata