Assessing Complex Emergency Management with Clinical Case-Vignettes: A Validation Study

OBJECTIVE: To evaluate whether responses to dynamic case-vignettes accurately reflect actual practices in complex emergency situations. We hypothesized that when obstetricians were faced with vignette of emergency situation identical to one they previously managed, they would report the management strategy they actually used. On the other hand, there is no reason to suppose that their response to a vignette based on a source case managed by another obstetrician would be the same as the actual management. METHODS: A multicenter vignette-based study was used in 7 French maternity units. We chose the example of severe postpartum hemorrhage (PPH) to study the use of case-vignettes for assessing the management of complex situations. We developed dynamic case-vignettes describing incidents of PPH in several steps, using documentation in patient files. Vignettes described the postpartum course and included multiple-choice questions detailing proposed clinical care. Each participating obstetrician was asked to evaluate 4 case-vignettes: 2 directly derived from cases they previously managed and 2 derived from other obstetricians’ cases. We compared the final treatment decision in vignette responses to those documented in the source-case by the overall agreement and the Kappa coefficient, both for the cases the obstetricians previously managed and the cases of others. RESULTS: Thirty obstetricians participated. Overall agreement between final treatment decisions in case-vignettes and documented care for cases obstetricians previously managed was 82% (Kappa coefficient: 0.75, 95% CI [0.62–0.88]). Overall agreement between final treatment decisions in case-vignettes and documented care in vignettes derived from other obstetricians’ cases was only 48% (Kappa coefficient: 0.30, 95% CI [0.12–0.48]). Final agreement with documented care was significantly better for cases based on their own previous cases than for others (p<0.001). CONCLUSIONS: Dynamic case-vignettes accurately reflect actual practices in complex emergency situations. Therefore, they can be used to assess the quality of management in these situations.