Error and Timeliness Analysis for Using Machine Learning to Predict Asthma Hospital Visits: Retrospective Cohort Study

BACKGROUND: Asthma hospital visits, including emergency department visits and inpatient stays, are a significant burden on health care. To leverage preventive care more effectively in managing asthma, we previously employed machine learning and data from the University of Washington Medicine (UWM) to build the world’s most accurate model to forecast which asthma patients will have asthma hospital visits during the following 12 months. OBJECTIVE: Currently, two questions remain regarding our model’s performance. First, for a patient who will have asthma hospital visits in the future, how far in advance can our model make an initial identification of risk? Second, if our model erroneously predicts a patient to have asthma hospital visits at the UWM during the following 12 months, how likely will the patient have ≥1 asthma hospital visit somewhere else or ≥1 surrogate indicator of a poor outcome? This work aims to answer these two questions. METHODS: Our patient cohort included every adult asthma patient who received care at the UWM between 2011 and 2018. Using the UWM data, our model made predictions on the asthma patients in 2018. For every such patient with ≥1 asthma hospital visit at the UWM in 2019, we computed the number of days in advance that our model gave an initial warning. For every such patient erroneously predicted to have ≥1 asthma hospital visit at the UWM in 2019, we used PreManage and the UWM data to check whether the patient had ≥1 asthma hospital visit outside of the UWM in 2019 or any surrogate indicators of poor outcomes. Such surrogate indicators included a prescription for systemic corticosteroids during the following 12 months, any type of visit for asthma exacerbation during the following 12 months, and asthma hospital visits between 13 and 24 months later. RESULTS: Among the 218 asthma patients in 2018 with asthma hospital visits at the UWM in 2019, 61.9% (135/218) were given initial warnings of such visits ≥3 months ahead by our model and 84.4% (184/218) were given initial warnings ≥1 day ahead. Among the 1310 asthma patients in 2018 who were erroneously predicted to have asthma hospital visits at the UWM in 2019, 29.01% (380/1310) had asthma hospital visits outside of the UWM in 2019 or surrogate indicators of poor outcomes. CONCLUSIONS: Our model gave timely risk warnings for most asthma patients with poor outcomes. We found that 29.01% (380/1310) of asthma patients for whom our model gave false-positive predictions had asthma hospital visits somewhere else during the following 12 months or surrogate indicators of poor outcomes, and thus were reasonable candidates for preventive interventions. There is still significant room for improving our model to give more accurate and more timely risk warnings. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): RR2-10.2196/5039