A multi-step approach to developing a health system evaluation framework for community-based health care

BACKGROUND: Community-based health care (CBHC) is a shift towards healthcare integration and community services closer to home. Variation in system approaches harkens the need for a conceptual framework to evaluate outcomes and impacts. We set out to develop a CBHC-specific evaluation framework in the context of a provincial ministry of health planning process in Canada. METHODS: A multi-step approach was used to develop the CBHC evaluation framework. Modified Delphi informed conceptualization and prioritization of indicators. Formative research identified evaluation framework elements (triple aim, global measures, and impact), health system levels (tiers), and potential CBHC indicators (n = 461). Two Delphi rounds were held. Round 1, panelists independently ranked indicators on CBHC relevance and health system tiering. Results were analyzed by coding agreement/disagreement frequency and central tendency measures. Round 2, a consensus meeting was used to discuss disagreement, identify Tier 1 indicators and concepts, and define indicators not relevant to CBHC (Tier 4). Post-Delphi, indicators and concepts were refined, Tier 1 concepts mapped to the evaluation framework, and indicator narratives developed. Three stakeholder consultations (scientific, government, and public/patient communities) were held for endorsement and recommendation. RESULTS: Round 1 Delphi results showed agreement for 300 and disagreement for 161 indicators. Round 2 consensus resulted in 103 top tier indicators (Tier 1 = 19, Tier 2 = 84), 358 bottom Tier 3 and 4 indicators, non-CBHC measure definitions, and eight Tier 1 indicator concepts—Mortality/Suicide; Quality of Life, and Patient Reported Outcome Measures; Global Patient Reported Experience Measures; Cost of Care, Access to Integrated Primary Care; Avoidable Emergency Department Use; Avoidable Hospitalization; and E-health Penetration. Post Delphi results refined Tier 3 (n = 289) and 4 (n = 69) indicators, and identified 18 Tier 2 and 3 concepts. When mapped to the evaluation framework, Tier 1 concepts showed full coverage across the elements. ‘Indicator narratives’ depicted systemness and integration for evaluating CBHC. Stakeholder consultations affirmed endorsement of the approach and evaluation framework; refined concepts; and provided key considerations to further operationalize and contextualize indicators, and evaluate CBHC as a health system approach. CONCLUSIONS: This research produced a novel evaluation framework to conceptualize and evaluate CBHC initiatives. The evaluation framework revealed the importance of a health system approach for evaluating CBHC.