OSCE rater cognition – an international multi-centre qualitative study

INTRODUCTION: This study aimed to explore the decision-making processes of raters during objective structured clinical examinations (OSCEs), in particular to explore the tacit assumptions and beliefs of raters as well as rater idiosyncrasies. METHODS: Thinking aloud protocol interviews were used to gather data on the thoughts of examiners during their decision-making, while watching trigger OSCE videos and rating candidates. A purposeful recruiting strategy was taken, with a view to interviewing both examiners with many years of experience (greater than six years) and those with less experience examining at final medical examination level. RESULTS: Thirty-one interviews were conducted in three centres in three different countries. Three themes were identified during data analysis, entitled ‘OSCEs are inauthentic’, ‘looking for glimpses of truth’ and ‘evolution with experience’. CONCLUSION: Raters perceive that the shortcomings of OSCEs can have unwanted effects on student behaviour. Some examiners, more likely the more experienced group, may deviate from an organisations directions due to perceived shortcomings of the assessment. No method of assessment is without flaw, and it is important to be aware of the limitations and shortcomings of assessment methods on student performance and examiner perception. Further study of assessor and student perception of OSCE performance would be helpful.