Single-rater reliability of a three-dimensional instrument for decision-making in tertiary triage and ICU- prioritization—a case vignette simulation study

Disconcerting reports from different EU countries during the first wave of the COVID-19 pandemic demonstrated the demand for supporting decision instruments and recommendations in case tertiary triage is needed. COVID-19 patients mainly present sequentially, not parallelly, and therefore ex-post triage scenarios were expected to be more likely than ex-ante ones. Decision-makers in these scenarios may be highly susceptible to second victim and moral injury effects, so that reliable and ethically justifiable algorithms would have been needed in case of overwhelming critical cases. To gather basic information about a potential tertiary triage instrument, we designed a three-dimensional instrument developed by an expert group using the Delphi technique. The instrument focused on three parameters: 1) estimated chance of survival, 2) estimated prognosis of regaining autonomy after treatment, and 3) estimated length of stay in the ICU. To validate and test the instrument, we conducted an anonymous online survey in 5 German hospitals addressing physicians that would have been in charge of decision-making in the case of a mass infection incident. Of about 80 physicians addressed, 47 responded. They were presented with 16 fictional ICU case vignettes (including 3 doublets) which they had to score using the three parameters of the instrument. We detected a good construct validity (Cronbach’s Alpha 0.735) and intra-reliability (p < 0.001, Cohens Kappa 0.497 to 0.574), but a low inter-reliability (p < 0.001, Cohen’s Kappa 0.252 to 0.327) for the three parameters. The best inter-reliability was detected for the estimated length of stay in the ICU. Further analysis revealed concerns in assessing the prognosis of the potentially remaining autonomy, especially in patients with only physical impairment. In accordance with German recommendations, we concluded that single-rater triage (which might happen in stressful and highly resource-limited situations) should be avoided to ensure patient and health care provider safety. Future work should concentrate on reliable and valid group decision instruments and algorithms and question whether the chance of survival as a single triage parameter should be complemented with other parameters, such as the estimated length of stay in the ICU. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1186/s12871-023-02173-2.