Comparison of the validity of checklist assessment in cardiac arrest simulations with an app in an academic hospital in Taiwan: a retrospective observational study

INTRODUCTION: Robust assessment is a crucial component in Advanced Cardiac Life Support (ACLS) training to determine whether participants have achieved learning objectives with little or no variation in their overall outcomes. This study aimed to evaluate resuscitation performance by real-time logs. We hypothesised that instructors may not be able to evaluate time-sensitive parameters, namely, chest compression fraction, time to initiating chest compression and time to initiating defibrillation accurately in a subjective manner. METHODS: Video records and formal checklist-based test results of Megacode scenarios for the ACLS certification examination at several hospitals in Taipei were examined. For the study interest, three time-sensitive parameters were measured via video review assisted by a mobile phone application, and were used for evaluation. We evaluated if the pass/fail results made by instructors via checklists were correlated with these parameters. RESULTS: A total of 185 Megacode scenarios were eligible for the final analysis. Among the three parameters, good chest compression fraction was statistically significant with a higher OR of passing (OR=3.65; 95% CI 1.36 to 9.91; p=0.01). In 112 participants with one parameter that did not meet the criteria, 25 were graded as fail, making the specificity 22.3% (95% CI 15.0% to 31.2%). CONCLUSIONS: Visual observation of cardiopulmonary resuscitation performance is not accurate when evaluating time-sensitive parameters. Objective results should be offered for training outcome evaluation, and also for feedback to participants.