Development of an assessment tool to measure communication skills among family medicine residents in the context of electronic medical record use

BACKGROUND: The introduction of the electronic medical record (EMR) has led to new communication skills that need to be taught and assessed. There is scarce literature on validated instruments measuring electronic-specific communication skills. The aim is to develop an assessment checklist that assesses the general and EMR-specific communication skills and evaluates their content validity and reliability. METHODS: Using the SEGUE theoretical framework for communication skills, the assessment checklist items were developed by the Communication Skills Working Group (CSWG) at the family medicine department using a literature review about the positive and negative aspects of EMR use on physician-patient communication. A group of faculty members rated real resident-patient encounters on two occasions, three weeks apart. Patients were asked to fill out the Communication Assessment Tool (CAT) at the end of the encounter. RESULTS: A total of 8 residents agreed to participate in the research, with 21 clinical encounters recorded. The average total score was 65.2 ± 6.9 and 48.1 ± 9.5 for the developed scale and the CAT scale, respectively. The scale reliability was good, with a Cronbach alpha of 0.694. The test-retest reliability was 0.873, p < 0.0001. For the total score on the developed checklist, the intraclass correlation coefficient between raters (ICC) was 0.429 [0.030,0.665], p-value of 0.019. The level of agreement between any two raters on the cumulative score of the 5 subsections ranged from 0.506 (interpersonal skills) to 0.969 (end encounter). CONCLUSION: This checklist is a reliable and valid instrument that combines basic and EMR-related communication skills. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1186/s12909-023-04216-1.