Challenges in evaluating Welfare to Work policy interventions: would an RCT design have been the answer to all our problems?

BACKGROUND: UK policy direction for recipients of unemployment and sickness benefits is to support these people into employment by increasing 'into work' interventions. Although the main aim of associated interventions is to increase levels of employment, improved health is stated as a benefit, and a driver of these interventions. This is therefore a potentially important policy intervention with respect to health and health inequalities, and needs to be validated through rigorous impact evaluation. We attempted to evaluate the Pathways Advisory Service intervention which aims to provide employment support for Incapacity Benefit recipients, but encountered a number of challenges and barriers to evaluation. This paper explores the issues that arose in designing a suitable evaluation of the Pathways Advisory Service. DISCUSSION: The main issues that arose were that characteristics of the intervention lead to difficulties in defining a suitable comparison group; and governance restrictions such as uncertainty regarding ethical consent processes and data sharing between agencies for research. Some of these challenges threatened fundamentally to limit the validity of any experimental or quasi-experimental evaluation we could design - restricting recruitment, data collection and identification of an appropriate comparison group. Although a cluster randomised controlled trial design was ethically justified to evaluate the Pathways Advisory Service, this was not possible because the intervention was already being widely implemented. However, this would not have solved other barriers to evaluation. There is no obvious method to perform a controlled evaluation for interventions where only a small proportion of those eligible are exposed. Improved communication between policymakers and researchers, clarification of data sharing protocols and improved guidelines for ethics committees are tangible ways which may reduce the current obstacles to this and other similar evaluations of policy interventions which tackle key determinants of health. SUMMARY: The evaluation of social interventions is hampered by more than their suitability to randomisation. Data sharing, participant identification and recruitment problems are common to randomised and non-randomised evaluation designs. These issues require further attention if we are to learn from current social policy.