Ensuring fair, safe, and interpretable artificial intelligence-based prediction tools in a real-world oncological setting

BACKGROUND: Cancer patients often experience treatment-related symptoms which, if uncontrolled, may require emergency department admission. We developed models identifying breast or genitourinary cancer patients at the risk of attending emergency department (ED) within 30-days and demonstrated the development, validation, and proactive approach to in-production monitoring of an artificial intelligence-based predictive model during a 3-month simulated deployment at a cancer hospital in the United States. METHODS: We used routinely-collected electronic health record data to develop our predictive models. We evaluated models including a variational autoencoder k-nearest neighbors algorithm (VAE-kNN) and model behaviors with a sample containing 84,138 observations from 28,369 patients. We assessed the model during a 77-day production period exposure to live data using a proactively monitoring process with predefined metrics. RESULTS: Performance of the VAE-kNN algorithm is exceptional (Area under the receiver-operating characteristics, AUC = 0.80) and remains stable across demographic and disease groups over the production period (AUC 0.74–0.82). We can detect issues in data feeds using our monitoring process to create immediate insights into future model performance. CONCLUSIONS: Our algorithm demonstrates exceptional performance at predicting risk of 30-day ED visits. We confirm that model outputs are equitable and stable over time using a proactive monitoring approach.