External validation of a prognostic model for intensive care unit mortality: a retrospective study using the Ontario Critical Care Information System

PURPOSE: To externally validate an intensive care unit (ICU) mortality prediction model that was created using the Ontario Critical Care Information System (CCIS), which includes the Multiple Organ Dysfunction Score (MODS). METHODS: We applied the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) recommendations to a prospective longitudinal cohort of patients discharged between 1 July 2015 and 31 December 31 2016 from 90 adult level-3 critical care units in Ontario. We used multivariable logistic regression with measures of discrimination, calibration-in-the-large, calibration slope, and flexible calibration plots to compare prediction model performance of the entire data set and for each ICU subtype. RESULTS: Among 121,201 CCIS records with ICU mortality of 11.3%, the C-statistic for the validation data set was 0.805. The C-statistic ranged from 0.775 to 0.846 among the ICU subtypes. After intercept recalibration to adjust the baseline risk, the mean predicted risk of death matched actual ICU mortality. The calibration slope was close to 1 with all CCIS data and ICU subtypes of cardiovascular and community hospitals with low ventilation rates. Calibration slopes significantly less than 1 were found for ICUs in teaching hospitals and community hospitals with high ventilation rates whereas coronary care units had a calibration slope significantly higher than 1. Calibration plots revealed over-prediction in high risk groups to a varying degree across all cohorts. CONCLUSIONS: A risk prediction model primarily based on the MODS shows reproducibility and transportability after intercept recalibration. Risk adjusting models that use existing and feasible data collection can support performance measurement at the individual ICU level. ELECTRONIC SUPPLEMENTARY MATERIAL: The online version of this article (10.1007/s12630-020-01686-5) contains supplementary material, which is available to authorized users.