Using Predictive Analytics to Identify Children at High Risk of Defaulting From a Routine Immunization Program: Feasibility Study

BACKGROUND: Despite the availability of free routine immunizations in low- and middle-income countries, many children are not completely vaccinated, vaccinated late for age, or drop out from the course of the immunization schedule. Without the technology to model and visualize risk of large datasets, vaccinators and policy makers are unable to identify target groups and individuals at high risk of dropping out; thus default rates remain high, preventing universal immunization coverage. Predictive analytics algorithm leverages artificial intelligence and uses statistical modeling, machine learning, and multidimensional data mining to accurately identify children who are most likely to delay or miss their follow-up immunization visits. OBJECTIVE: This study aimed to conduct feasibility testing and validation of a predictive analytics algorithm to identify the children who are likely to default on subsequent immunization visits for any vaccine included in the routine immunization schedule. METHODS: The algorithm was developed using 47,554 longitudinal immunization records, which were classified into the training and validation cohorts. Four machine learning models (random forest; recursive partitioning; support vector machines, SVMs; and C-forest) were used to generate the algorithm that predicts the likelihood of each child defaulting from the follow-up immunization visit. The following variables were used in the models as predictors of defaulting: gender of the child, language spoken at the child’s house, place of residence of the child (town or city), enrollment vaccine, timeliness of vaccination, enrolling staff (vaccinator or others), date of birth (accurate or estimated), and age group of the child. The models were encapsulated in the predictive engine, which identified the most appropriate method to use in a given case. Each of the models was assessed in terms of accuracy, precision (positive predictive value), sensitivity, specificity and negative predictive value, and area under the curve (AUC). RESULTS: Out of 11,889 cases in the validation dataset, the random forest model correctly predicted 8994 cases, yielding 94.9% sensitivity and 54.9% specificity. The C-forest model, SVMs, and recursive partitioning models improved prediction by achieving 352, 376, and 389 correctly predicted cases, respectively, above the predictions made by the random forest model. All models had a C-statistic of 0.750 or above, whereas the highest statistic (AUC 0.791, 95% CI 0.784-0.798) was observed in the recursive partitioning algorithm. CONCLUSIONS: This feasibility study demonstrates that predictive analytics can accurately identify children who are at a higher risk for defaulting on follow-up immunization visits. Correct identification of potential defaulters opens a window for evidence-based targeted interventions in resource limited settings to achieve optimal immunization coverage and timeliness.