Grading reflective essays: the construct validity and reliability of a newly developed Tool- GRE-9

BACKGROUND: The main objective of this study is to assess the construct validity and reliability of the Grading Reflective Essays-9 (GRE-9). METHODS: This study took place in a major tertiary academic medical center in Beirut, Lebanon. 104 reflective essays written by years 1–3 residents in the department of Family Medicine at the American University of Beirut Medical Center (AUBMC) were graded by 2 trained raters who independently scored the essays using GRE-9. GRE-9 scores were then correlated with scores on communication skills OSCE stations and in-training examinations to investigate, respectively, convergent and divergent validity. One of the 2 raters scored the essays twice one month apart to assess the reliability of the GRE-9 using intra rater reliability and internal consistency. RESULTS: There was a weak, non-significant correlation between GRE-9 score and In training examination (ITE) score (r = − .213, p = .395). There was a moderate, non-significant correlation between GRE-9 scores and the Objective structured clinical examination (OSCE) communication station scores (r = − .412 p = .162). The correlation coefficient between trails 1 and 2 was significant (r = .832, p = .000). Intra class correlation coefficient (ICC) analysis demonstrated almost perfect intra-rater agreement (0.819; 95% CI: 0.741–0.875) of the test ratings over time. CONCLUSIONS: GRE-9, is a short, concise, easy-to-use reliable grading tool for reflective essays that has demonstrated moderate to substantial intra-rater reliability and evidence of divergent validity. The study found non-significant correlations between reflective writing scores OSCE communication scores demonstrating a lack of relationship between reflective writing and this measure of performance. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1186/s12909-023-04845-6.