Analysing 3429 digital supervisory interactions between Community Health Workers in Uganda and Kenya: the development, testing and validation of an open access predictive machine learning web app

BACKGROUND: Despite the growth in mobile technologies (mHealth) to support Community Health Worker (CHW) supervision, the nature of mHealth-facilitated supervision remains underexplored. One strategy to support supervision at scale could be artificial intelligence (AI) modalities, including machine learning. We developed an open access, machine learning web application (CHWsupervisor) to predictively code instant messages exchanged between CHWs based on supervisory interaction codes. We document the development and validation of the web app and report its predictive accuracy. METHODS: CHWsupervisor was developed using 2187 instant messages exchanged between CHWs and their supervisors in Uganda. The app was then validated on 1242 instant messages from a separate digital CHW supervisory network in Kenya. All messages from the training and validation data sets were manually coded by two independent human coders. The predictive performance of CHWsupervisor was determined by comparing the primary supervisory codes assigned by the web app, against those assigned by the human coders and calculating observed percentage agreement and Cohen’s kappa coefficients. RESULTS: Human inter-coder reliability for the primary supervisory category of messages across the training and validation datasets was ‘substantial’ to ‘almost perfect’, as suggested by observed percentage agreements of 88–95% and Cohen’s kappa values of 0.7–0.91. In comparison to the human coders, the predictive accuracy of the CHWsupervisor web app was ‘moderate’, suggested by observed percentage agreements of 73–78% and Cohen’s kappa values of 0.51–0.56. CONCLUSIONS: Augmenting human coding is challenging because of the complexity of supervisory exchanges, which often require nuanced interpretation. A realistic understanding of the potential of machine learning approaches should be kept in mind by practitioners, as although they hold promise, supportive supervision still requires a level of human expertise. Scaling-up digital CHW supervision may therefore prove challenging. Trial registration: This was not a clinical trial and was therefore not registered as such. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1186/s12960-021-00699-5.