Cargando…

Comparison of correctly and incorrectly classified patients for in-hospital mortality prediction in the intensive care unit

BACKGROUND: The use of machine learning is becoming increasingly popular in many disciplines, but there is still an implementation gap of machine learning models in clinical settings. Lack of trust in models is one of the issues that need to be addressed in an effort to close this gap. No models are...

Descripción completa

Detalles Bibliográficos
Autores principales: Stenwig, Eline, Salvi, Giampiero, Salvo Rossi, Pierluigi, Skjærvold, Nils Kristian
Formato: Online Artículo Texto
Lenguaje:English
Publicado: BioMed Central 2023
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10124049/
https://www.ncbi.nlm.nih.gov/pubmed/37095430
http://dx.doi.org/10.1186/s12874-023-01921-9
_version_ 1785029772293701632
author Stenwig, Eline
Salvi, Giampiero
Salvo Rossi, Pierluigi
Skjærvold, Nils Kristian
author_facet Stenwig, Eline
Salvi, Giampiero
Salvo Rossi, Pierluigi
Skjærvold, Nils Kristian
author_sort Stenwig, Eline
collection PubMed
description BACKGROUND: The use of machine learning is becoming increasingly popular in many disciplines, but there is still an implementation gap of machine learning models in clinical settings. Lack of trust in models is one of the issues that need to be addressed in an effort to close this gap. No models are perfect, and it is crucial to know in which use cases we can trust a model and for which cases it is less reliable. METHODS: Four different algorithms are trained on the eICU Collaborative Research Database using similar features as the APACHE IV severity-of-disease scoring system to predict hospital mortality in the ICU. The training and testing procedure is repeated 100 times on the same dataset to investigate whether predictions for single patients change with small changes in the models. Features are then analysed separately to investigate potential differences between patients consistently classified correctly and incorrectly. RESULTS: A total of 34 056 patients (58.4%) are classified as true negative, 6 527 patients (11.3%) as false positive, 3 984 patients (6.8%) as true positive, and 546 patients (0.9%) as false negatives. The remaining 13 108 patients (22.5%) are inconsistently classified across models and rounds. Histograms and distributions of feature values are compared visually to investigate differences between groups. CONCLUSIONS: It is impossible to distinguish the groups using single features alone. Considering a combination of features, the difference between the groups is clearer. Incorrectly classified patients have features more similar to patients with the same prediction rather than the same outcome. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1186/s12874-023-01921-9.
format Online
Article
Text
id pubmed-10124049
institution National Center for Biotechnology Information
language English
publishDate 2023
publisher BioMed Central
record_format MEDLINE/PubMed
spelling pubmed-101240492023-04-25 Comparison of correctly and incorrectly classified patients for in-hospital mortality prediction in the intensive care unit Stenwig, Eline Salvi, Giampiero Salvo Rossi, Pierluigi Skjærvold, Nils Kristian BMC Med Res Methodol Research BACKGROUND: The use of machine learning is becoming increasingly popular in many disciplines, but there is still an implementation gap of machine learning models in clinical settings. Lack of trust in models is one of the issues that need to be addressed in an effort to close this gap. No models are perfect, and it is crucial to know in which use cases we can trust a model and for which cases it is less reliable. METHODS: Four different algorithms are trained on the eICU Collaborative Research Database using similar features as the APACHE IV severity-of-disease scoring system to predict hospital mortality in the ICU. The training and testing procedure is repeated 100 times on the same dataset to investigate whether predictions for single patients change with small changes in the models. Features are then analysed separately to investigate potential differences between patients consistently classified correctly and incorrectly. RESULTS: A total of 34 056 patients (58.4%) are classified as true negative, 6 527 patients (11.3%) as false positive, 3 984 patients (6.8%) as true positive, and 546 patients (0.9%) as false negatives. The remaining 13 108 patients (22.5%) are inconsistently classified across models and rounds. Histograms and distributions of feature values are compared visually to investigate differences between groups. CONCLUSIONS: It is impossible to distinguish the groups using single features alone. Considering a combination of features, the difference between the groups is clearer. Incorrectly classified patients have features more similar to patients with the same prediction rather than the same outcome. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1186/s12874-023-01921-9. BioMed Central 2023-04-24 /pmc/articles/PMC10124049/ /pubmed/37095430 http://dx.doi.org/10.1186/s12874-023-01921-9 Text en © The Author(s) 2023 https://creativecommons.org/licenses/by/4.0/Open AccessThis article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ (https://creativecommons.org/licenses/by/4.0/) . The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/ (https://creativecommons.org/publicdomain/zero/1.0/) ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
spellingShingle Research
Stenwig, Eline
Salvi, Giampiero
Salvo Rossi, Pierluigi
Skjærvold, Nils Kristian
Comparison of correctly and incorrectly classified patients for in-hospital mortality prediction in the intensive care unit
title Comparison of correctly and incorrectly classified patients for in-hospital mortality prediction in the intensive care unit
title_full Comparison of correctly and incorrectly classified patients for in-hospital mortality prediction in the intensive care unit
title_fullStr Comparison of correctly and incorrectly classified patients for in-hospital mortality prediction in the intensive care unit
title_full_unstemmed Comparison of correctly and incorrectly classified patients for in-hospital mortality prediction in the intensive care unit
title_short Comparison of correctly and incorrectly classified patients for in-hospital mortality prediction in the intensive care unit
title_sort comparison of correctly and incorrectly classified patients for in-hospital mortality prediction in the intensive care unit
topic Research
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10124049/
https://www.ncbi.nlm.nih.gov/pubmed/37095430
http://dx.doi.org/10.1186/s12874-023-01921-9
work_keys_str_mv AT stenwigeline comparisonofcorrectlyandincorrectlyclassifiedpatientsforinhospitalmortalitypredictionintheintensivecareunit
AT salvigiampiero comparisonofcorrectlyandincorrectlyclassifiedpatientsforinhospitalmortalitypredictionintheintensivecareunit
AT salvorossipierluigi comparisonofcorrectlyandincorrectlyclassifiedpatientsforinhospitalmortalitypredictionintheintensivecareunit
AT skjærvoldnilskristian comparisonofcorrectlyandincorrectlyclassifiedpatientsforinhospitalmortalitypredictionintheintensivecareunit