How important is evaluator's alignment for the reliability of assessment procedure for recognizing the good practice?

Implementing evidence-based interventions that are proven to be successful and effective in improving individual, community, and population health is widely recognized as the best approach to addressing public health challenges. To avoid the development and implementation of less effective or successful or even harmful practices, clear criteria for the assessment of practices that are considering different dimensions of the interventions in public health, are needed. Registers of programs and interventions in the field of prevention and health promotion play an important role here, provided that appropriate evaluation/assessment criteria are used for the selection of practices presented on the portals. Nevertheless, the challenge of choosing the “right” approach and criteria in assessing the health promotion and disease prevention interventions still remain. The development of the system for the assessment of health promotion and disease prevention interventions consists of several steps, from defining the criteria, development of evaluation methodology, selection of evaluators, piloting the assessment procedure and finally to regular use of the entire system/portal. An often-overlooked, but important step is how the evaluators understand the criteria and methodology and if it is properly used in assessing the interventions. In this workshop we will present how we used statistical tools to assess the consistency between the evaluators and thereby gaining insight into the reliability of the criteria as a measuring instrument for the assessment of the interventions as well as to check how the evaluators understand the criteria and methodology and if it is properly used in assessing the interventions.