Evaluation of the quality of multiple-choice questions according to the students’ academic level

BACKGROUND: One of the most important challenges in medical education is the preparation of multiple-choice questions able to discriminate between students with different academic level. Average questions may be very easy for students with good performance, reducing their discriminant power in this group of students. The aim of this study was to analyze if the discriminative power of multiple-choice questions is different according to the students’ academic performance. METHODS: We retrospectively analyzed the difficulty and discrimination indices of 257 multiple-choice questions used for the end of course examination of pathophysiology and analyzed whether the discrimination indices were lower in students with good academic performance (group 1) than in students with moderate/poor academic performance (group 2). We also evaluated whether case-based questions maintained their discriminant power better than factual questions in both groups of students or not. Comparison of the difficulty and discrimination indices between both groups was based on the Wilcoxon test. RESULTS: Difficulty index was significantly higher in group 1 (median: 0.78 versus 0.56; P < 0.001) and discrimination index was significantly higher in group 2 (median: 0.21 versus 0.28; P < 0.001). Factual questions had higher discriminative indices in group 2 than in group 1 (median: 0.28 versus 0.20; P < 0.001), but discriminative indices of case-based questions did not differ significantly between groups (median: 0.30 versus 0.24; P = 0.296). CONCLUSIONS: Multiple-choice question exams have lower discriminative power in the group of students with high scores. The use of clinical vignettes may allow to maintain the discriminative power of multiple-choice questions.