Application of a Machine Learning Algorithm to Develop and Validate a Prediction Model for Ambulatory Non-Arrivals

BACKGROUND: Non-arrivals to scheduled ambulatory visits are common and lead to a discontinuity of care, poor health outcomes, and increased subsequent healthcare utilization. Reducing non-arrivals is important given their association with poorer health outcomes and cost to health systems. OBJECTIVE: To develop and validate a prediction model for ambulatory non-arrivals. DESIGN: Retrospective cohort study. PATIENTS OR SUBJECTS: Patients at an integrated health system who had an outpatient visit scheduled from January 1, 2020, to February 28, 2022. MAIN MEASURES: Non-arrivals to scheduled appointments. KEY RESULTS: There were over 4.3 million ambulatory appointments from 1.2 million adult patients. Patients with appointment non-arrivals were more likely to be single, racial/ethnic minorities, and not having an established primary care provider compared to those who arrived at their appointments. A prediction model using the XGBoost machine learning algorithm had the highest AUC value (0.768 [0.767–0.770]). Using SHAP values, the most impactful features in the model include rescheduled appointments, lead time (number of days from scheduled to appointment date), appointment provider, number of days since last appointment with the same department, and a patient’s prior appointment status within the same department. Scheduling visits close to an appointment date is predicted to be less likely to result in a non-arrival. Overall, the prediction model calibrated well for each department, especially over the operationally relevant probability range of 0 to 40%. Departments with fewer observations and lower non-arrival rates generally had a worse calibration. CONCLUSIONS: Using a machine learning algorithm, we developed a prediction model for non-arrivals to scheduled ambulatory appointments usable for all medical specialties. The proposed prediction model can be deployed within an electronic health system or integrated into other dashboards to reduce non-arrivals. Future work will focus on the implementation and application of the model to reduce non-arrivals. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1007/s11606-023-08065-y.