Applying a validated scoring rubric to pre-clerkship medical students’ standardized patient notes: a pilot study

BACKGROUND: With the elimination in 2021 of the United States Medical Licensing Examination Step 2 Clinical Skills test, it is incumbent upon U.S. medical schools to develop local validated assessments of clinical reasoning. While much attention has been paid to summative exams for graduating students, formative exams for pre-clerkship students have not been well studied. METHODS: We applied the University of Illinois at Chicago College of Medicine (UIC-COM) Patient Note (PN) Scoring Rubric to templated PNs written by 103 pre-clerkship students for two cases in an objective structured clinical examination (OSCE) at the Yale School of Medicine. The rubric consists of four section scores (Documentation, Differential Diagnosis, Justification, and Workup, each scored 1 to 4) and a composite score (scaled 23 to 100). We calculated item discrimination for each section score and Cronbach’s alpha for each case. We surveyed students about their experience writing the templated PN. RESULTS: Mean Documentation, Differential Diagnosis, Justification, Workup, and composite scores for case A were 2.16, 1.80, 1.65, 2.29, and 47.67, respectively. For case B, the scores were 2.13, 1.21, 1.60, 1.67, and 40.54, respectively. Item discrimination ranged from 0.41 to 0.80. Cronbach’s alpha for cases A and B was 0.48 and 0.25, respectively. A majority of the students felt that the exercise was useful and appropriate to their level of training. CONCLUSIONS: Despite performing poorly, pre-clerkship students found the note-writing task beneficial. Reliability of the scoring rubric was suboptimal, and modifications are needed to make this exercise a suitable measure of clinical reasoning. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1186/s12909-023-04424-9.