Boosting for insight and/or boosting for agency? How to maximize accurate test interpretation with natural frequencies

BACKGROUND: Many physicians do not know how to accurately interpret test results using Bayes’ rule. As a remedy, two kinds of interventions have been shown effective: boosting insight and boosting agency with natural frequencies. To boost insight, test statistics are provided in natural frequencies (rather than conditional probabilities), without instructions on how to use them. To boost agency, a training is provided on how to translate probabilities into natural frequencies and apply them in Bayes’ rule. What has not been shown is whether boosting agency is sufficient or if representing test statistics in natural frequencies may additionally boost insight to maximize accurate test interpretation. METHODS: We used a pre/posttest design to assess test interpretation accuracy of 577 medical students before and after a training on two Bayesian reasoning tasks, one providing conditional probabilities, the other natural frequencies. The pretest assessed baseline abilities versus the effect of natural frequencies to boost insight. After participants received a training on how to translate conditional probabilities into natural frequencies and how to apply them in Bayes’ rule, test interpretation skills were assessed using the same tasks again, comparing the effects of training-induced agency with versus without additionally boosting insight (i.e., test statistics in natural frequencies versus conditional probabilities). RESULTS: Compared to the test question formatted in conditional probabilities (34% correct answers), natural frequencies facilitated Bayesian reasoning without training (68%), that is, they increased insight. The training on how to use natural frequencies improved performance for tasks formatted in conditional probabilities (64%). Performance was maximal after training and with test statistics formatted in natural frequencies, that is, with a combination of boosting insight and agency (89%). CONCLUSIONS: Natural frequencies should be used to boost insight and agency to maximize effective use of teaching resources. Thus, mandating that test statistics are provided in natural frequencies and adopting short trainings on how to translate conditional probabilities into natural frequencies and how to apply them in Bayes’ rule will help to maximize accurate test interpretation. TRIAL REGISTRATION: The study was a registered with the German Clinical Trial Registry (DRKS00008723; 06/03/2015).