Cargando…
The Effect of Outcome Selection on the Performance of Prediction Models in Patients at Risk for Sepsis
Risk prediction models for patients with suspected sepsis have been derived on and applied to various outcomes, including readily available outcomes such as hospital mortality and ICU admission as well as longer-term mortality outcomes that may be more important to patients. It is unknown how select...
Autores principales: | , , , , , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Wolters Kluwer Health
2020
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7063898/ https://www.ncbi.nlm.nih.gov/pubmed/32166298 http://dx.doi.org/10.1097/CCE.0000000000000078 |
_version_ | 1783504779062804480 |
---|---|
author | Taylor, Stephanie P. Chou, Shih-Hsiung McWilliams, Andrew D. Russo, Mark Heffner, Alan C. Murphy, Stephanie Evans, Susan L. Rossman, Whitney Kowalkowski, Marc |
author_facet | Taylor, Stephanie P. Chou, Shih-Hsiung McWilliams, Andrew D. Russo, Mark Heffner, Alan C. Murphy, Stephanie Evans, Susan L. Rossman, Whitney Kowalkowski, Marc |
author_sort | Taylor, Stephanie P. |
collection | PubMed |
description | Risk prediction models for patients with suspected sepsis have been derived on and applied to various outcomes, including readily available outcomes such as hospital mortality and ICU admission as well as longer-term mortality outcomes that may be more important to patients. It is unknown how selecting different outcomes influences model performance in patients at risk for sepsis. OBJECTIVES: Evaluate the impact of outcome selection on risk model performance and weighting of individual predictor variables. DESIGN, SETTING, AND PARTICIPANTS: We retrospectively analyzed adults hospitalized with suspected infection from January 2014 to September 2017 at 12 hospitals. MAIN OUTCOMES AND MEASURES: We used routinely collected clinical data to derive logistic regression models for four outcomes: hospital mortality, composite ICU length of stay greater than 72 hours or hospital mortality, 30-day mortality, and 90-day mortality. We compared the performance of the models using area under the receiver operating characteristic curve and calibration plots. RESULTS: Among 52,184 admissions, 2,030 (4%) experienced hospital mortality, 6,659 (13%) experienced the composite of hospital mortality or ICU length of stay greater than 72 hours, 3,417 (7%) experienced 30-day mortality, and 5,655 (11%) experienced 90-day mortality. Area under the receiver operating characteristic curves decreased when hospital-based models were applied to predict 30-day (hospital mortality = 0.88–0.85; –0.03, composite ICU length of stay greater than 72 hours or hospital mortality = 0.90–0.81; –0.09) and 90-day mortality (hospital mortality = 0.88–0.81; –0.07, composite ICU length of stay greater than 72 hours or hospital mortality = 0.90–0.76; –0.14; all p < 0.01). Models were well calibrated for derived (root-mean-square error = 5–15) but not alternate outcomes (root-mean-square error = 8–35). CONCLUSIONS AND RELEVANCE: Risk models trained to predict readily available hospital-based outcomes in suspected sepsis show poorer discrimination and calibration when applied to 30- and 90-day mortality. Interpretation and application of risk models for patients at risk of sepsis should consider these findings. |
format | Online Article Text |
id | pubmed-7063898 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2020 |
publisher | Wolters Kluwer Health |
record_format | MEDLINE/PubMed |
spelling | pubmed-70638982020-03-12 The Effect of Outcome Selection on the Performance of Prediction Models in Patients at Risk for Sepsis Taylor, Stephanie P. Chou, Shih-Hsiung McWilliams, Andrew D. Russo, Mark Heffner, Alan C. Murphy, Stephanie Evans, Susan L. Rossman, Whitney Kowalkowski, Marc Crit Care Explor Observational Studies Risk prediction models for patients with suspected sepsis have been derived on and applied to various outcomes, including readily available outcomes such as hospital mortality and ICU admission as well as longer-term mortality outcomes that may be more important to patients. It is unknown how selecting different outcomes influences model performance in patients at risk for sepsis. OBJECTIVES: Evaluate the impact of outcome selection on risk model performance and weighting of individual predictor variables. DESIGN, SETTING, AND PARTICIPANTS: We retrospectively analyzed adults hospitalized with suspected infection from January 2014 to September 2017 at 12 hospitals. MAIN OUTCOMES AND MEASURES: We used routinely collected clinical data to derive logistic regression models for four outcomes: hospital mortality, composite ICU length of stay greater than 72 hours or hospital mortality, 30-day mortality, and 90-day mortality. We compared the performance of the models using area under the receiver operating characteristic curve and calibration plots. RESULTS: Among 52,184 admissions, 2,030 (4%) experienced hospital mortality, 6,659 (13%) experienced the composite of hospital mortality or ICU length of stay greater than 72 hours, 3,417 (7%) experienced 30-day mortality, and 5,655 (11%) experienced 90-day mortality. Area under the receiver operating characteristic curves decreased when hospital-based models were applied to predict 30-day (hospital mortality = 0.88–0.85; –0.03, composite ICU length of stay greater than 72 hours or hospital mortality = 0.90–0.81; –0.09) and 90-day mortality (hospital mortality = 0.88–0.81; –0.07, composite ICU length of stay greater than 72 hours or hospital mortality = 0.90–0.76; –0.14; all p < 0.01). Models were well calibrated for derived (root-mean-square error = 5–15) but not alternate outcomes (root-mean-square error = 8–35). CONCLUSIONS AND RELEVANCE: Risk models trained to predict readily available hospital-based outcomes in suspected sepsis show poorer discrimination and calibration when applied to 30- and 90-day mortality. Interpretation and application of risk models for patients at risk of sepsis should consider these findings. Wolters Kluwer Health 2020-01-24 /pmc/articles/PMC7063898/ /pubmed/32166298 http://dx.doi.org/10.1097/CCE.0000000000000078 Text en Copyright © 2020 The Authors. Published by Wolters Kluwer Health, Inc. on behalf of the Society of Critical Care Medicine. https://creativecommons.org/licenses/by-nc-nd/4.0/This is an open-access article distributed under the terms of the Creative Commons Attribution-Non Commercial-No Derivatives License 4.0 (CCBY-NC-ND) (https://creativecommons.org/licenses/by-nc-nd/4.0/) , where it is permissible to download and share the work provided it is properly cited. The work cannot be changed in any way or used commercially without permission from the journal. |
spellingShingle | Observational Studies Taylor, Stephanie P. Chou, Shih-Hsiung McWilliams, Andrew D. Russo, Mark Heffner, Alan C. Murphy, Stephanie Evans, Susan L. Rossman, Whitney Kowalkowski, Marc The Effect of Outcome Selection on the Performance of Prediction Models in Patients at Risk for Sepsis |
title | The Effect of Outcome Selection on the Performance of Prediction Models in Patients at Risk for Sepsis |
title_full | The Effect of Outcome Selection on the Performance of Prediction Models in Patients at Risk for Sepsis |
title_fullStr | The Effect of Outcome Selection on the Performance of Prediction Models in Patients at Risk for Sepsis |
title_full_unstemmed | The Effect of Outcome Selection on the Performance of Prediction Models in Patients at Risk for Sepsis |
title_short | The Effect of Outcome Selection on the Performance of Prediction Models in Patients at Risk for Sepsis |
title_sort | effect of outcome selection on the performance of prediction models in patients at risk for sepsis |
topic | Observational Studies |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7063898/ https://www.ncbi.nlm.nih.gov/pubmed/32166298 http://dx.doi.org/10.1097/CCE.0000000000000078 |
work_keys_str_mv | AT taylorstephaniep theeffectofoutcomeselectionontheperformanceofpredictionmodelsinpatientsatriskforsepsis AT choushihhsiung theeffectofoutcomeselectionontheperformanceofpredictionmodelsinpatientsatriskforsepsis AT mcwilliamsandrewd theeffectofoutcomeselectionontheperformanceofpredictionmodelsinpatientsatriskforsepsis AT russomark theeffectofoutcomeselectionontheperformanceofpredictionmodelsinpatientsatriskforsepsis AT heffneralanc theeffectofoutcomeselectionontheperformanceofpredictionmodelsinpatientsatriskforsepsis AT murphystephanie theeffectofoutcomeselectionontheperformanceofpredictionmodelsinpatientsatriskforsepsis AT evanssusanl theeffectofoutcomeselectionontheperformanceofpredictionmodelsinpatientsatriskforsepsis AT rossmanwhitney theeffectofoutcomeselectionontheperformanceofpredictionmodelsinpatientsatriskforsepsis AT kowalkowskimarc theeffectofoutcomeselectionontheperformanceofpredictionmodelsinpatientsatriskforsepsis AT taylorstephaniep effectofoutcomeselectionontheperformanceofpredictionmodelsinpatientsatriskforsepsis AT choushihhsiung effectofoutcomeselectionontheperformanceofpredictionmodelsinpatientsatriskforsepsis AT mcwilliamsandrewd effectofoutcomeselectionontheperformanceofpredictionmodelsinpatientsatriskforsepsis AT russomark effectofoutcomeselectionontheperformanceofpredictionmodelsinpatientsatriskforsepsis AT heffneralanc effectofoutcomeselectionontheperformanceofpredictionmodelsinpatientsatriskforsepsis AT murphystephanie effectofoutcomeselectionontheperformanceofpredictionmodelsinpatientsatriskforsepsis AT evanssusanl effectofoutcomeselectionontheperformanceofpredictionmodelsinpatientsatriskforsepsis AT rossmanwhitney effectofoutcomeselectionontheperformanceofpredictionmodelsinpatientsatriskforsepsis AT kowalkowskimarc effectofoutcomeselectionontheperformanceofpredictionmodelsinpatientsatriskforsepsis |