How to choose an evidence-based medicine knowledge test for medical students? Comparison of three knowledge measures

BACKGROUND: There are a few studies of alignment between different knowledge-indices for evidence-based medicine (EBM). The aim of this study was to investigate whether the type of test used to assess knowledge of EBM affects the estimation of this knowledge in medical students. METHODS: Medical students enrolled in 1-week EBM course were tested with the Fresno, Berlin, and ACE tests at the beginning and the end of the course. We evaluated the ability of these tests to detect a change in the acquired level of EBM knowledge and compared the estimates of change with those of the Control group that was tested with the ACE and Berlin tests before and after an unrelated non-EBM course. The distributions of test scores and average item difficulty indices were compared among the tests and the groups. RESULTS: Test scores improved on all three tests when compared with their pre-test results and the control. Students had on average a “good” performance on the ACE test, “sufficient” performance on the Berlin test, and “insufficient” performance or have “not passed” on the Fresno test. The post-test improvements in performance on the Fresno test (median 31% increase in percent scores, 95% confidence interval (CI) 25–42%) outperformed those on the ACE (13, 95% CI 13–20%) and Berlin tests (13, 95% CI 7–20%). Post-test score distributions demonstrated that the ACE test had less potential to discriminate between levels of EBM knowledge than other tests. CONCLUSION: The use of different EBM tests resulted in different assessment of general EBM knowledge in a sample of graduate medical students, with lowest results on the Fresno and highest on the ACE test. In the light of these findings, EBM knowledge assessment should be based on the course’s content and learning objectives. ELECTRONIC SUPPLEMENTARY MATERIAL: The online version of this article (10.1186/s12909-018-1391-z) contains supplementary material, which is available to authorized users.