Using theory of change frameworks to develop evaluation strategies for research engagement: results of a pre‐pilot study

INTRODUCTION: Inadequate community and stakeholder engagement can lead to accusations that research is unethical and can delay or slow research or translation of results to practice. Such experiences have led major funders as well as regulatory and advisory bodies to establish minimal requirements for community and stakeholder engagement in HIV and other clinical research. However, systematic efforts to formally evaluate the contributions and impact of particular practices are lacking. METHODS: A theory of change framework aligned with Good Participatory Practice for TB clinical trials was used to develop a set of measures for use in a minimally burdensome survey of trial implementing sites. The survey was pre‐piloted with three TB trial sites in North America, South America and Asia to assess the feasibility of surveying global research sites in a systematic way, and to see if the measures captured informative variation in the use of engagement strategies and desired outcomes. Surveys were conducted at baseline and six months. In‐depth interviews were conducted with site staff prior to the baseline survey to understand how sites conceptualized the concepts underlying the framework and the extent to which they viewed their work as aligned with the framework. RESULTS: Survey measures captured considerable variability in the intensity and variety of engagement strategies, both across sites and within sites over time, and moderate variability in outcomes. Interviews indicated that underlying concepts were often unfamiliar to staff at baseline, but the goals of engagement aligned well with existing values. CONCLUSIONS: Brief, targeted surveys of trial sites to characterize use of broad strategies, specific practices and some outcomes are a feasible option for evaluating good participatory practice. Additional testing is warranted to assess and enhance validity, reliability and predictive value of indicators. Options for collecting outcome measures through additional objective means should be explored.