Retrospective Observational Study of the Clinical Performance Characteristics of a Machine Learning Approach to Early Sepsis Identification

To estimate performance characteristics and impact on care processes of a machine learning, early sepsis recognition tool embedded in the electronic medical record. DESIGN: Retrospective review of electronic medical records and outcomes to determine sepsis prevalence among patients about whom a warning was received in real time and timing of that warning compared with clinician recognition of potential sepsis as determined by actions documented in the electronic medical record. SETTING: Acute care, nonteaching hospital. PATIENTS: Patients in the emergency department, observation unit, and adult inpatient care units who had sepsis diagnosed either by clinical codes or by Center for Medicare and Medicaid Services Severe Sepsis and Septic Shock: Management Bundle (SEP-1) criteria for severe sepsis and patients who had machine learning–generated advisories about a high risk of sepsis. INTERVENTIONS: Noninterventional study. MEASUREMENTS AND MAIN RESULTS: Using two different definitions of sepsis as “true” sepsis, we measured the sensitivity and early warning clinical utility. Using coded sepsis to define true positives, we measured the positive predictive value of the early warnings. Sensitivity was 28.6% and 43.6% for coded sepsis and severe sepsis, respectively. The positive predictive value of an alert was 37.9% for coded sepsis. Clinical utility (true positive and earlier advisory than clinical recognition) was 2.2% and 1.6% for the two different definitions of sepsis. Use of the tool did not improve sepsis mortality rates. CONCLUSIONS: Performance characteristics were different than previously described in this retrospective assessment of real-time warnings. Real-world testing of retrospectively validated models is essential. The early warning clinical utility may vary depending on a hospital’s state of sepsis readiness and embrace of sepsis order bundles.