The reliability and usability of the Anesthesiologists’ Non-Technical Skills (ANTS) system in simulation research

BACKGROUND: Non-technical skills (NTS) such as leadership and team work are important in providing good quality of care. One system to assess physicians’ NTS is the Anesthesiologists’ Non-Technical Skills (ANTS) system. The present study evaluates the ANTS system on the interrater reliability and usability for research purposes. METHODS: Ten anesthesiologists and 20 anesthesiology residents performed two resuscitation scenarios (with and without the presence of distractors) in a simulation room with a full-scale patient simulator. The scenarios were videotaped. Two independent raters rated the NTS of the anesthesiologists using the ANTS system. The intraclass correlation coefficients (ICC) were calculated to determine the interrater reliability of both the total NTS score and the measured differences between the two scenarios. The raters filled out a questionnaire to obtain insights in the usability of the ANTS system for research purposes. RESULTS: The ICC for the total score of the NTS was substantial (0.683), and the ICC of the elements varied between 0.371 for assessing capabilities and 0.670 for providing and maintaining standards. The intraclass correlation coefficient of measuring differences was fair (0.502). The raters judged the usability as good. CONCLUSIONS: The ANTS system was reliable for the total score and usable to measure physicians’ NTS in a research setting. However, there was variation between the reliability of the elements. We recommend that if the ANTS is used for research, a pilot study should determine elements not applicable or observable in the scenario of interest; these elements should be excluded from the study. ELECTRONIC SUPPLEMENTARY MATERIAL: The online version of this article (doi:10.1186/s41077-016-0013-2) contains supplementary material, which is available to authorized users.