Systematic approach to evaluating and confirming the utility of a suite of national health system performance (HSP) indicators in Canada: a modified Delphi study

OBJECTIVES: Evaluating an existing suite of health system performance (HSP) indicators for continued reporting using a systematic criteria-based assessment and national consensus conference. DESIGN: Modified Delphi approach with technical and leadership groups, an online survey of stakeholders and convening a national consensus conference. SETTING: A national health information steward, the Canadian Institute for Health Information (CIHI). PARTICIPANTS: A total of 73 participants, comprised 61 conference attendants/stakeholders from across Canada and 12 national health information steward staff. PRIMARY AND SECONDARY OUTCOME MEASURES: Indicator dispositions of retention, additional stakeholder consultation, further redevelopment or retirement. RESULTS: 4 dimensions (usability, importance, scientific soundness and feasibility) typically used to select measures for reporting were expanded to 18 criteria grouped under the 4 dimensions through a process of research and testing. Definitions for each criterion were developed and piloted. Once the definitions were established, 56 of CIHI's publicly reported HSP indicators were evaluated against the criteria using modified Delphi approaches. Of the 56 HSP indicators evaluated, 9 measures were ratified for retirement, 7 were identified for additional consultation and 3 for further research and development. A pre-Consensus Conference survey soliciting feedback from stakeholders on indicator recommendations received 48 responses (response rate of 79%). CONCLUSIONS: A systematic evaluation of HSP indicators informed the development of objective recommendations for continued reporting. The evaluation was a fruitful exercise to identify technical considerations for calculating indicators, furthering our understanding of how measures are used by stakeholders, as well as harmonising actions that could be taken to ensure relevancy, reduce indicator chaos and build consensus with stakeholders.