Proof of impact and pipeline planning: directions and challenges for social audit in the health sector

Social audits are typically observational studies, combining qualitative and quantitative uptake of evidence with consultative interpretation of results. This often falters on issues of causality because their cross-sectional design limits interpretation of time relations and separation out of other indirect associations. Social audits drawing on methods of randomised controlled cluster trials (RCCT) allow more certainty about causality. Randomisation means that exposure occurs independently of all events that precede it – it converts potential confounders and other covariates into random differences. In 2008, CIET social audits introduced randomisation of the knowledge translation component with subsequent measurement of impact in the changes introduced. This “proof of impact” generates an additional layer of evidence in a cost-effective way, providing implementation-ready solutions for planners. Pipeline planning is a social audit that incorporates stepped wedge RCCTs. From a listing of districts/communities as a sampling frame, individual entities (communities, towns, districts) are randomly assigned to waves of intervention. Measurement of the impact takes advantage of the delay occasioned by the reality that there are insufficient resources to implement everywhere at the same time. The impact in the first wave contrasts with the second wave, which in turn contrasts with a third wave, and so on until all have received the intervention. Provided care is taken to achieve reasonable balance in the random allocation of communities, towns or districts to the waves, the resulting analysis can be straightforward. Where there is sufficient management interest in and commitment to evidence, pipeline planning can be integrated in the roll-out of programmes where real time information can improve the pipeline. Not all interventions can be randomly allocated, however, and random differences can still distort measurement. Other issues include contamination of the subsequent waves, ambiguity of indicators, “participant effects” that result from lack of blinding and lack of placebos, ethics and, not least important, the skills to do pipeline planning correctly.