Quantifying uncertainty in intervention effectiveness with structured expert judgement: an application to obstetric fistula

OBJECTIVE: To demonstrate a new application of structured expert judgement to assess the effectiveness of surgery to correct obstetric fistula in a low-income setting. Intervention effectiveness is a major input of evidence-informed priority setting in healthcare, but information on intervention effectiveness is generally lacking. This is particularly problematic in the context of poorly resourced healthcare settings where even efficacious interventions fail to translate into improvements in health. The few intervention effectiveness studies related to obstetric fistula treatment focus on the experience of single facilities and do not consider the impact of multiple factors that may affect health outcomes. DESIGN: We use the classical model of structured expert judgement, a method that has been used to quantify uncertainty in the areas of engineering and environmental risk assessment when data are unavailable. Under this method, experts quantify their uncertainty about rates of long-term disability in patients with fistula following treatment in different contexts, but the information content drawn from their responses is statistically conditioned on the accuracy and informativeness of their responses to a set of calibration questions. Through this method, we develop best estimates and uncertainty bounds for the rate of disability associated with each treatment scenario and setting. PARTICIPANTS: Eight experts in obstetric fistula repair in low and middle income countries. RESULTS: Estimates developed using performance weights were statistically superior to those involving a simple averaging of expert responses. The performance-weight decision maker's assessments are narrower for 9 of the 10 calibration questions and 21 of 23 variables of interest. CONCLUSIONS: We find that structured expert judgement is a viable approach to investigating the effectiveness of medical interventions where randomised controlled trials are not possible. Understanding the effectiveness of surgery performed at different types of facilities can guide programme planning to increase access to fistula treatment.