What can we learn from studying control arms of randomised VAW prevention intervention evaluations: reflections on expected measurement error, meaningful change and the utility of RCTs

Background: Randomised controlled trials (RCTs) are a gold standard for evaluations in public health, economics and social sciences, including prevention of violence against women (VAW). They substantially reduce bias, but do not eliminate measurement error. Control arms often show change, but this is rarely systematically examined. Objective: We present a secondary analysis of data from the control arms of evaluations of VAW prevention programming to understand measurement variance over time, factors that may systematically impact this and make recommendations for stronger trial design and interpretation. Methods: We examine data from six RCTs and one quasi-experimental study, all of which used comparable measures. We look at change over time among control participants in prevalence of physical intimate partner violence (IPV), sexual IPV, and severe physical/sexual IPV, by participants’ gender and study design (cohort vs. repeat cross-sectional). Results: On average, repeated assessments of past year IPV varied by 3.21 (95%Cis 1.59,4.83) percentage points for the studies with no active control arms. The prevalence at endline, as a proportion of that at baseline, on average differed by 17.7%. In 10/35 assessments from 4/7 studies, the difference was more than 30%. We did not find evidence of the Hawthorne effect or repeat interview bias as explanations. Our findings largely supported non-differential misclassification (measurement error) as the most likely error and it was a greater problem for men. Conclusions: Control arms are very valuable, but in VAW research their measures fluctuate. This must be considered in sample size calculations. We need more rigorous criteria for determining trial effect. Our findings suggest this may be an absolute change in prevalence of 7% and proportionate change of 0.4 or more (especially for studies in populations with lower IPV prevalence (<20%)). More elaborate pre-defined outcomes are necessary for determining impact (or possible harms) of VAW prevention interventions.