Measuring competency-relevant knowledge in the competency-oriented student progress test

Background: Since 2013 a competency-oriented student progress test (SKPT) has been administered at a number of German medical schools. The questions are generated on the basis of a two-dimensional blueprint, on which one axis contains the five competency domains – communicative competence (CO), practical clinical competence (CP), theoretical clinical competence (CT), scientific competence (SC), and professional decision-making competence (PR) – that form part of the competency model of the National Competency-based Catalogue of Learning Objectives for Undergraduate Medicine (NKLM). The feedback for students is structured in part according to these domains. The aim of our study is to examine if 1. the results differentiated by competency domain show adequate measurement accuracy and 2. if the results for the different domains also contain different information. Methods: The SKPTs for the years 2013 to 2017, taken by a total of 3,027 students, were examined. The measurement accuracy was determined using the coefficient glb (greatest bound to reliability) and the standard error of measurement; discriminant analysis of the principal components was carried out to demonstrate differentiation between the competency domains. Results: The reliability of the competency domains was above 0.8 for all SKPTs; exceptions to this were seen in two of the tests for CO and PR that had a reliability of 0.7–0.8. The results for all of the individual competency domains differed in their informational content compared to the overall of the other domains; the same applies for all pairwise comparisons, with the exceptions of CP and CT. Discussion: The SKPT feedback for students that is differentiated by competency domains basically fulfills the requirements for measurement reliability and distinctness. An improvement of the measurement quality for CO and PR and a better differentiation between CP and CT is desirable.