A Comparison of Checklist and Domain-Based Ratings in the Assessment of Objective Structured Clinical Examination (OSCE) Performance

Introduction: The Objective Structured Clinical Examination (OSCE) is the mainstay of clinical assessment in the final-year undergraduate Family Medicine clerkship at King Faisal Specialist Hospital and Research Centre (KFSHRC). The gold standard for OSCE assessment is the checklist rating, completed by physician examiners. Numerous studies have suggested that global or domain-based OSCE ratings may be a better indicator of competence than checklist ratings. The aim of this study was to examine the utility of domain-based OSCE ratings in the context of final-year, undergraduate, Family Medicine OSCE examinations in Riyadh, Saudi Arabia. This is akin to an exercise in quality improvement, as we continuously look for ways to improve our OSCE assessment processes. Methods: This study utilised a quantitative methodology. Three final year OSCE exams were chosen. Physicians rated each student using a checklist score and using a more holistic domain-based score. Physician checklist scores and physician domain-based scores were then compared, and correlation was assessed. We also looked at the internal consistency of the scoring methods. Results: A significant correlation was found between checklist and domain-based scores by physicians for all exams (r=0.858, p<0.01), with a good internal consistency for these methodologies for all exams. Conclusion: The results demonstrate that both checklist and domain-based scores offer some benefit to the assessment, with a similar internal consistency and strong correlation. Domain-based ratings should be utilised for softer skills that are not easily assessed by checklists. There is clearly a need to rethink our OSCE assessment. The assessment should combine checklist and domain-based physician scores. As trainees become more experienced, checklist OSCE may penalise directness and efficiency, while domain-based ratings would offer a better appraisal of competence, and have been shown to be more sensitive to the level of training and expertise. Changing the assessment methods will lead to necessary changes in the student approach to the OSCE and improve authenticity and validity.