Cargando…

Early warning score validation methodologies and performance metrics: a systematic review

BACKGROUND: Early warning scores (EWS) have been developed as clinical prognostication tools to identify acutely deteriorating patients. In the past few years, there has been a proliferation of studies that describe the development and validation of novel machine learning-based EWS. Systematic revie...

Descripción completa

Detalles Bibliográficos
Autores principales: Fang, Andrew Hao Sen, Lim, Wan Tin, Balakrishnan, Tharmmambal
Formato: Online Artículo Texto
Lenguaje:English
Publicado: BioMed Central 2020
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7301346/
https://www.ncbi.nlm.nih.gov/pubmed/32552702
http://dx.doi.org/10.1186/s12911-020-01144-8
_version_ 1783547670864855040
author Fang, Andrew Hao Sen
Lim, Wan Tin
Balakrishnan, Tharmmambal
author_facet Fang, Andrew Hao Sen
Lim, Wan Tin
Balakrishnan, Tharmmambal
author_sort Fang, Andrew Hao Sen
collection PubMed
description BACKGROUND: Early warning scores (EWS) have been developed as clinical prognostication tools to identify acutely deteriorating patients. In the past few years, there has been a proliferation of studies that describe the development and validation of novel machine learning-based EWS. Systematic reviews of published studies which focus on evaluating performance of both well-established and novel EWS have shown conflicting conclusions. A possible reason is the heterogeneity in validation methods applied. In this review, we aim to examine the methodologies and metrics used in studies which perform EWS validation. METHODS: A systematic review of all eligible studies from the MEDLINE database and other sources, was performed. Studies were eligible if they performed validation on at least one EWS and reported associations between EWS scores and inpatient mortality, intensive care unit (ICU) transfers, or cardiac arrest (CA) of adults. Two reviewers independently did a full-text review and performed data abstraction by using standardized data-worksheet based on the TRIPOD (Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis) checklist. Meta-analysis was not performed due to heterogeneity. RESULTS: The key differences in validation methodologies identified were (1) validation dataset used, (2) outcomes of interest, (3) case definition, time of EWS use and aggregation methods, and (4) handling of missing values. In terms of case definition, among the 48 eligible studies, 34 used the patient episode case definition while 12 used the observation set case definition, and 2 did the validation using both case definitions. Of those that used the patient episode case definition, 18 studies validated the EWS at a single point of time, mostly using the first recorded observation. The review also found more than 10 different performance metrics reported among the studies. CONCLUSIONS: Methodologies and performance metrics used in studies performing validation on EWS were heterogeneous hence making it difficult to interpret and compare EWS performance. Standardizing EWS validation methodology and reporting can potentially address this issue.
format Online
Article
Text
id pubmed-7301346
institution National Center for Biotechnology Information
language English
publishDate 2020
publisher BioMed Central
record_format MEDLINE/PubMed
spelling pubmed-73013462020-06-18 Early warning score validation methodologies and performance metrics: a systematic review Fang, Andrew Hao Sen Lim, Wan Tin Balakrishnan, Tharmmambal BMC Med Inform Decis Mak Research Article BACKGROUND: Early warning scores (EWS) have been developed as clinical prognostication tools to identify acutely deteriorating patients. In the past few years, there has been a proliferation of studies that describe the development and validation of novel machine learning-based EWS. Systematic reviews of published studies which focus on evaluating performance of both well-established and novel EWS have shown conflicting conclusions. A possible reason is the heterogeneity in validation methods applied. In this review, we aim to examine the methodologies and metrics used in studies which perform EWS validation. METHODS: A systematic review of all eligible studies from the MEDLINE database and other sources, was performed. Studies were eligible if they performed validation on at least one EWS and reported associations between EWS scores and inpatient mortality, intensive care unit (ICU) transfers, or cardiac arrest (CA) of adults. Two reviewers independently did a full-text review and performed data abstraction by using standardized data-worksheet based on the TRIPOD (Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis) checklist. Meta-analysis was not performed due to heterogeneity. RESULTS: The key differences in validation methodologies identified were (1) validation dataset used, (2) outcomes of interest, (3) case definition, time of EWS use and aggregation methods, and (4) handling of missing values. In terms of case definition, among the 48 eligible studies, 34 used the patient episode case definition while 12 used the observation set case definition, and 2 did the validation using both case definitions. Of those that used the patient episode case definition, 18 studies validated the EWS at a single point of time, mostly using the first recorded observation. The review also found more than 10 different performance metrics reported among the studies. CONCLUSIONS: Methodologies and performance metrics used in studies performing validation on EWS were heterogeneous hence making it difficult to interpret and compare EWS performance. Standardizing EWS validation methodology and reporting can potentially address this issue. BioMed Central 2020-06-18 /pmc/articles/PMC7301346/ /pubmed/32552702 http://dx.doi.org/10.1186/s12911-020-01144-8 Text en © The Author(s) 2020 Open AccessThis article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
spellingShingle Research Article
Fang, Andrew Hao Sen
Lim, Wan Tin
Balakrishnan, Tharmmambal
Early warning score validation methodologies and performance metrics: a systematic review
title Early warning score validation methodologies and performance metrics: a systematic review
title_full Early warning score validation methodologies and performance metrics: a systematic review
title_fullStr Early warning score validation methodologies and performance metrics: a systematic review
title_full_unstemmed Early warning score validation methodologies and performance metrics: a systematic review
title_short Early warning score validation methodologies and performance metrics: a systematic review
title_sort early warning score validation methodologies and performance metrics: a systematic review
topic Research Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7301346/
https://www.ncbi.nlm.nih.gov/pubmed/32552702
http://dx.doi.org/10.1186/s12911-020-01144-8
work_keys_str_mv AT fangandrewhaosen earlywarningscorevalidationmethodologiesandperformancemetricsasystematicreview
AT limwantin earlywarningscorevalidationmethodologiesandperformancemetricsasystematicreview
AT balakrishnantharmmambal earlywarningscorevalidationmethodologiesandperformancemetricsasystematicreview