Developing Script Concordance Test Items in Otolaryngology to Improve Clinical Reasoning Skills: Validation using Consensus Analysis and Psychometrics

BACKGROUND: Script concordance testing is widely practiced to foster and assess clinical reasoning. Our study aimed to develop script concordance test (SCT) in the specialty of otolaryngology and test the validation using panel response pattern and consensus index. MATERIALS AND METHODS: The methodology was an evolving pattern of constructing SCTs, administering them to the panel members, and optimizing the panel with response patterns and consensus index. The SCT’s final items were chosen to be administered to the students. RESULTS: We developed 98 items of SCT and administered them to 20 panel members. The mean score of the panel members for these 98 items was 79.5 (standard deviation [SD] = 4.4). The consensus index calculated for the 98-item SCT ranged from 25.81 to 100. Sixteen items had bimodal and uniform response patterns; the consensus index improved when eliminated. We administered the rest 82 items of SCT to 30 undergraduate and ten postgraduate students. The mean score of undergraduate students was 61.1 (SD = 7.5) and that of postgraduate students was 67.7 (SD = 6.3). Cronbach’s alpha for the 82-item SCT was 0.74. Excluding the 22 poor items, the final SCT instrument of 60 items had a Cronbach’s alpha of 0.82. CONCLUSION: Our study revealed that a consensus index above 60 had a good item-total correlation and be used to optimize the items for panel responses in SCT, necessitating further studies on this aspect. Our study also revealed that the panel response clustering pattern could be used to categorize the items, although bimodal and uniform distribution patterns need further differentiation.