Cargando…

Validating a Machine Learning Algorithm to Predict 30-Day Re-Admissions in Patients With Heart Failure: Protocol for a Prospective Cohort Study

BACKGROUND: Big data solutions, particularly machine learning predictive algorithms, have demonstrated the ability to unlock value from data in real time in many settings outside of health care. Rapid growth in electronic medical record adoption and the shift from a volume-based to a value-based rei...

Descripción completa

Detalles Bibliográficos
Autores principales: Kakarmath, Sujay, Golas, Sara, Felsted, Jennifer, Kvedar, Joseph, Jethwani, Kamal, Agboola, Stephen
Formato: Online Artículo Texto
Lenguaje:English
Publicado: JMIR Publications 2018
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6231891/
https://www.ncbi.nlm.nih.gov/pubmed/30181113
http://dx.doi.org/10.2196/resprot.9466
_version_ 1783370322487017472
author Kakarmath, Sujay
Golas, Sara
Felsted, Jennifer
Kvedar, Joseph
Jethwani, Kamal
Agboola, Stephen
author_facet Kakarmath, Sujay
Golas, Sara
Felsted, Jennifer
Kvedar, Joseph
Jethwani, Kamal
Agboola, Stephen
author_sort Kakarmath, Sujay
collection PubMed
description BACKGROUND: Big data solutions, particularly machine learning predictive algorithms, have demonstrated the ability to unlock value from data in real time in many settings outside of health care. Rapid growth in electronic medical record adoption and the shift from a volume-based to a value-based reimbursement structure in the US health care system has spurred investments in machine learning solutions. Machine learning methods can be used to build flexible, customized, and automated predictive models to optimize resource allocation and improve the efficiency and quality of health care. However, these models are prone to the problems of overfitting, confounding, and decay in predictive performance over time. It is, therefore, necessary to evaluate machine learning–based predictive models in an independent dataset before they can be adopted in the clinical practice. In this paper, we describe the protocol for independent, prospective validation of a machine learning–based model trained to predict the risk of 30-day re-admission in patients with heart failure. OBJECTIVE: This study aims to prospectively validate a machine learning–based predictive model for inpatient admissions in patients with heart failure by comparing its predictions of risk for 30-day re-admissions against outcomes observed prospectively in an independent patient cohort. METHODS: All adult patients with heart failure who are discharged alive from an inpatient admission will be prospectively monitored for 30-day re-admissions through reports generated by the electronic medical record system. Of these, patients who are part of the training dataset will be excluded to avoid information leakage to the algorithm. An expected sample size of 1228 index admissions will be required to observe a minimum of 100 30-day re-admission events. Deidentified structured and unstructured data will be fed to the algorithm, and its prediction will be recorded. The overall model performance will be assessed using the concordance statistic. Furthermore, multiple discrimination thresholds for screening high-risk patients will be evaluated according to the sensitivity, specificity, predictive values, and estimated cost savings to our health care system. RESULTS: The project received funding in April 2017 and data collection began in June 2017. Enrollment was completed in July 2017. Data analysis is currently underway, and the first results are expected to be submitted for publication in October 2018. CONCLUSIONS: To the best of our knowledge, this is one of the first studies to prospectively evaluate a predictive machine learning algorithm in a real-world setting. Findings from this study will help to measure the robustness of predictions made by machine learning algorithms and set a realistic benchmark for expectations of gains that can be made through its application to health care. REGISTERED REPORT IDENTIFIER: RR1-10.2196/9466
format Online
Article
Text
id pubmed-6231891
institution National Center for Biotechnology Information
language English
publishDate 2018
publisher JMIR Publications
record_format MEDLINE/PubMed
spelling pubmed-62318912018-12-10 Validating a Machine Learning Algorithm to Predict 30-Day Re-Admissions in Patients With Heart Failure: Protocol for a Prospective Cohort Study Kakarmath, Sujay Golas, Sara Felsted, Jennifer Kvedar, Joseph Jethwani, Kamal Agboola, Stephen JMIR Res Protoc Protocol BACKGROUND: Big data solutions, particularly machine learning predictive algorithms, have demonstrated the ability to unlock value from data in real time in many settings outside of health care. Rapid growth in electronic medical record adoption and the shift from a volume-based to a value-based reimbursement structure in the US health care system has spurred investments in machine learning solutions. Machine learning methods can be used to build flexible, customized, and automated predictive models to optimize resource allocation and improve the efficiency and quality of health care. However, these models are prone to the problems of overfitting, confounding, and decay in predictive performance over time. It is, therefore, necessary to evaluate machine learning–based predictive models in an independent dataset before they can be adopted in the clinical practice. In this paper, we describe the protocol for independent, prospective validation of a machine learning–based model trained to predict the risk of 30-day re-admission in patients with heart failure. OBJECTIVE: This study aims to prospectively validate a machine learning–based predictive model for inpatient admissions in patients with heart failure by comparing its predictions of risk for 30-day re-admissions against outcomes observed prospectively in an independent patient cohort. METHODS: All adult patients with heart failure who are discharged alive from an inpatient admission will be prospectively monitored for 30-day re-admissions through reports generated by the electronic medical record system. Of these, patients who are part of the training dataset will be excluded to avoid information leakage to the algorithm. An expected sample size of 1228 index admissions will be required to observe a minimum of 100 30-day re-admission events. Deidentified structured and unstructured data will be fed to the algorithm, and its prediction will be recorded. The overall model performance will be assessed using the concordance statistic. Furthermore, multiple discrimination thresholds for screening high-risk patients will be evaluated according to the sensitivity, specificity, predictive values, and estimated cost savings to our health care system. RESULTS: The project received funding in April 2017 and data collection began in June 2017. Enrollment was completed in July 2017. Data analysis is currently underway, and the first results are expected to be submitted for publication in October 2018. CONCLUSIONS: To the best of our knowledge, this is one of the first studies to prospectively evaluate a predictive machine learning algorithm in a real-world setting. Findings from this study will help to measure the robustness of predictions made by machine learning algorithms and set a realistic benchmark for expectations of gains that can be made through its application to health care. REGISTERED REPORT IDENTIFIER: RR1-10.2196/9466 JMIR Publications 2018-09-04 /pmc/articles/PMC6231891/ /pubmed/30181113 http://dx.doi.org/10.2196/resprot.9466 Text en ©Sujay Kakarmath, Sara Golas, Jennifer Felsted, Joseph Kvedar, Kamal Jethwani, Stephen Agboola. Originally published in JMIR Research Protocols (http://www.researchprotocols.org), 04.09.2018. https://creativecommons.org/licenses/by/4.0/This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on http://www.researchprotocols.org, as well as this copyright and license information must be included.
spellingShingle Protocol
Kakarmath, Sujay
Golas, Sara
Felsted, Jennifer
Kvedar, Joseph
Jethwani, Kamal
Agboola, Stephen
Validating a Machine Learning Algorithm to Predict 30-Day Re-Admissions in Patients With Heart Failure: Protocol for a Prospective Cohort Study
title Validating a Machine Learning Algorithm to Predict 30-Day Re-Admissions in Patients With Heart Failure: Protocol for a Prospective Cohort Study
title_full Validating a Machine Learning Algorithm to Predict 30-Day Re-Admissions in Patients With Heart Failure: Protocol for a Prospective Cohort Study
title_fullStr Validating a Machine Learning Algorithm to Predict 30-Day Re-Admissions in Patients With Heart Failure: Protocol for a Prospective Cohort Study
title_full_unstemmed Validating a Machine Learning Algorithm to Predict 30-Day Re-Admissions in Patients With Heart Failure: Protocol for a Prospective Cohort Study
title_short Validating a Machine Learning Algorithm to Predict 30-Day Re-Admissions in Patients With Heart Failure: Protocol for a Prospective Cohort Study
title_sort validating a machine learning algorithm to predict 30-day re-admissions in patients with heart failure: protocol for a prospective cohort study
topic Protocol
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6231891/
https://www.ncbi.nlm.nih.gov/pubmed/30181113
http://dx.doi.org/10.2196/resprot.9466
work_keys_str_mv AT kakarmathsujay validatingamachinelearningalgorithmtopredict30dayreadmissionsinpatientswithheartfailureprotocolforaprospectivecohortstudy
AT golassara validatingamachinelearningalgorithmtopredict30dayreadmissionsinpatientswithheartfailureprotocolforaprospectivecohortstudy
AT felstedjennifer validatingamachinelearningalgorithmtopredict30dayreadmissionsinpatientswithheartfailureprotocolforaprospectivecohortstudy
AT kvedarjoseph validatingamachinelearningalgorithmtopredict30dayreadmissionsinpatientswithheartfailureprotocolforaprospectivecohortstudy
AT jethwanikamal validatingamachinelearningalgorithmtopredict30dayreadmissionsinpatientswithheartfailureprotocolforaprospectivecohortstudy
AT agboolastephen validatingamachinelearningalgorithmtopredict30dayreadmissionsinpatientswithheartfailureprotocolforaprospectivecohortstudy