Application of objective structured clinical examination (OSCE) for the evaluation of Kampo medicine training

BACKGROUND: The purpose of this study was to develop an objective, content-valid, and reliable assessment method for Kampo medicine using an objective structured clinical examination (OSCE) for the assessment of clinical competence in Kampo medicine. METHODS: We developed a blueprint followed by a list of 47 assessment items and three task scenarios related to clinical competence in Kampo medicine. An eight-member test committee checked the relevance of the assessment items on a Likert scale. We calculated a content validity index and content validity ratio, and used the Angoff method to set the passing threshold. We trained a total of nine simulated patients with three assigned to each scenario. We conducted an OSCE for 11 candidates with varying medical abilities, and conducted three stations per person, which were evaluated by one evaluator in one room by direct observation. We used video recordings to test the inter-rater reliability of the three raters. We used the test results to verify the reliability of the assessment chart. RESULTS: The inter-rater reliability (intraclass correlation coefficient [2,1]) was 0.973. The reliability of the assessment chart for each scenario (Cronbach’s α) was 0.86, 0.89, and 0.85 for Scenarios 1, 2, and 3, respectively. The reliability of the assessment chart for the whole OSCE (Cronbach’s α) was 0.90. CONCLUSIONS: We developed a content-valid new OSCE assessment method for Kampo medicine and obtained high inter-rater and test reliabilities. Our findings suggest that this is one of the most reliable evaluation methods for assessing clinical competence in Kampo medicine. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1186/s12909-022-03264-3.