Measuring antenatal counseling skill with a milestone-based assessment tool: a validation study

BACKGROUND: Antenatal counseling for parents in the setting of expected preterm delivery is an important component of pediatric training. However, healthcare professionals receive a variable amount and quality of formal training. This study evaluated and discussed validity of a practical tool to assess antenatal counseling skills and provide evaluative feedback: the Antenatal Counseling Milestones Scale (ACoMS). METHODS: Experts in antenatal counseling developed an anchored milestone-based tool to evaluate observable skills. Study participants with a range of antenatal counseling skills were recruited to participate in simulation of counseling sessions in person or via video with standardized patient actors presenting with preterm labor at 23 weeks’ gestation. Two faculty observers scored each session independently using the ACoMS. Participants completed an ACoMS self-assessment, demographic, and feedback survey. Validity was measured with weighted kappas for inter-rater agreement, Kruskal–Wallis and Dunn’s tests for milestone levels between degrees of expertise in counseling, and cronbach’s alpha for item consistency. RESULTS: Forty-two participants completed observed counseling sessions. Of the 17 items included in the tool, 15 items were statistically significant with scores scaling with level of training. A majority of elements had fair-moderate agreement between raters, and there was high internal consistency amongst all items. CONCLUSION: This study demonstrates that the internal structure of the ACoMS rubric has greater than fair inter-rater reliability and high internal consistency amongst items. Content validity is supported by the scale’s ability to discern level of training. Application of the ACoMS to clinical encounters is needed to determine utility in clinical practice. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1186/s12909-023-04282-5.