Human judgement forecasting tournaments: A feasibility study based on the COVID-19 pandemic with public health practitioners in England()

OBJECTIVES: To explore the viability of running human judgement forecasting tournaments with public health practitioners, and to gather initial data on forecasting accuracy and participant perceptions of forecasting. STUDY DESIGN: Quality improvement study comprising two COVID-19 forecasting tournaments using Brier Skill Score scoring and a follow-up participant questionnaire. METHODS: Over two forecasting tournaments, public health registrars in the East Midlands, UK, assigned probabilities to future possible binary events relating to COVID-19. Participants also completed a questionnaire on their experiences of forecasting. RESULTS: There were 17 participants in the first tournament and nine in the second tournament, with no new participants. In both tournaments, the majority of participants scored a Brier Skill Score above the benchmark of 0. The median Brier Skill Score improved slightly between the two tournaments. Participants reported luck and changing political climates as impacting their performance. Participants reported forecasting in their day job but had received no formal training to do so. CONCLUSIONS: Forecasting is an important public health skill, and human judgement forecasting tournaments can be run amongst public health practitioners with little time and resource requirements. Further research would help identify whether training, teamwork or other interventions can improve public health forecasting accuracy.