Use of simulation scenarios and vote cards in teaching critical appraisal concepts in evidence-based medicine

BACKGROUND: The most effective method of teaching critical appraisal concepts remains unclear. We used simulation scenarios in a Risk-of-Bias (RoB) 2.0 framework to teach the various biases that may affect randomized controlled trials and assessed whether including this interactive session in an evidence-based medicine (EBM) course for third-year preclinical medical students can optimize their understanding of critical appraisal concepts. METHODS: The session had 13 modules, each corresponding to a particular risk of bias in RoB 2.0. Each module included a simulated scenario, followed by data presentation and a generalized conclusion. The students were subsequently asked to use colored vote cards to indicate whether they agreed, had some concern, or disagreed with the conclusion and to justify their answers. On the basis of the students’ answers, the facilitator debriefed the scenario and addressed the specific bias. In each module, the students were required to demonstrate critical thinking in analyzing the claims and quality of the supporting evidence and in justifying their decisions, thus conceptualizing their understanding of research biases. RESULTS: We included 306 students across two pilot sessions in spring 2020 and 2021, and the response rate was 97.4%. The students were least able to discern the following problems: baseline imbalances when assessing allocation bias (correct answers: 9.06%), missing outcome data when assessing attrition bias (correct answers: 11.65%), and balanced nonprotocol interventions when assessing performance bias (correct answers: 14.88%). The postcourse survey revealed several aspects of the interactive session that the students appreciated or found challenging. CONCLUSION: Preclinical medical students generally appreciated the inclusion of simulation scenarios and vote cards in an EBM course. The use of vote cards facilitated medical students’ understanding of critical appraisal concepts, uncovered areas that they found challenging to understand, and encouraged their active participation. Such interactive sessions should be increasingly included in medical education. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1186/s12909-023-04738-8.