Measures of outer setting constructs for implementation research: A systematic review and analysis of psychometric quality

BACKGROUND: Despite their influence, outer setting barriers (e.g., policies, financing) are an infrequent focus of implementation research. The objective of this systematic review was to identify and assess the psychometric properties of measures of outer setting used in behavioral and mental health research. METHODS: Data collection involved (a) search string generation, (b) title and abstract screening, (c) full-text review, (d) construct mapping, and (e) measure forward searches. Outer setting constructs were defined using the Consolidated Framework for Implementation Research (CFIR). The search strategy included four relevant constructs separately: (a) cosmopolitanism, (b) external policy and incentives, (c) patient needs and resources, and (d) peer pressure. Information was coded using nine psychometric criteria: (a) internal consistency, (b) convergent validity, (c) discriminant validity, (d) known-groups validity, (e) predictive validity, (f) concurrent validity, (g) structural validity, (h) responsiveness, and (i) norms. Frequencies were calculated to summarize the availability of psychometric information. Information quality was rated using a 5-point scale and a final median score was calculated for each measure. RESULTS: Systematic searches yielded 20 measures: four measures of the general outer setting domain, seven of cosmopolitanism, four of external policy and incentives, four of patient needs and resources, and one measure of peer pressure. Most were subscales within full scales assessing implementation context. Typically, scales or subscales did not have any psychometric information available. Where information was available, the quality was most often rated as “1-minimal” or “2-adequate.” CONCLUSION: To our knowledge, this is the first systematic review to focus exclusively on measures of outer setting factors used in behavioral and mental health research and comprehensively assess a range of psychometric criteria. The results highlight the limited quantity and quality of measures at this level. Researchers should not assume “one size fits all” when measuring outer setting constructs. Some outer setting constructs may be more appropriately and efficiently assessed using objective indices or administrative data reflective of the system rather than the individual.