Cargando…
Comparison of correctly and incorrectly classified patients for in-hospital mortality prediction in the intensive care unit
BACKGROUND: The use of machine learning is becoming increasingly popular in many disciplines, but there is still an implementation gap of machine learning models in clinical settings. Lack of trust in models is one of the issues that need to be addressed in an effort to close this gap. No models are...
Autores principales: | , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
BioMed Central
2023
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10124049/ https://www.ncbi.nlm.nih.gov/pubmed/37095430 http://dx.doi.org/10.1186/s12874-023-01921-9 |
_version_ | 1785029772293701632 |
---|---|
author | Stenwig, Eline Salvi, Giampiero Salvo Rossi, Pierluigi Skjærvold, Nils Kristian |
author_facet | Stenwig, Eline Salvi, Giampiero Salvo Rossi, Pierluigi Skjærvold, Nils Kristian |
author_sort | Stenwig, Eline |
collection | PubMed |
description | BACKGROUND: The use of machine learning is becoming increasingly popular in many disciplines, but there is still an implementation gap of machine learning models in clinical settings. Lack of trust in models is one of the issues that need to be addressed in an effort to close this gap. No models are perfect, and it is crucial to know in which use cases we can trust a model and for which cases it is less reliable. METHODS: Four different algorithms are trained on the eICU Collaborative Research Database using similar features as the APACHE IV severity-of-disease scoring system to predict hospital mortality in the ICU. The training and testing procedure is repeated 100 times on the same dataset to investigate whether predictions for single patients change with small changes in the models. Features are then analysed separately to investigate potential differences between patients consistently classified correctly and incorrectly. RESULTS: A total of 34 056 patients (58.4%) are classified as true negative, 6 527 patients (11.3%) as false positive, 3 984 patients (6.8%) as true positive, and 546 patients (0.9%) as false negatives. The remaining 13 108 patients (22.5%) are inconsistently classified across models and rounds. Histograms and distributions of feature values are compared visually to investigate differences between groups. CONCLUSIONS: It is impossible to distinguish the groups using single features alone. Considering a combination of features, the difference between the groups is clearer. Incorrectly classified patients have features more similar to patients with the same prediction rather than the same outcome. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1186/s12874-023-01921-9. |
format | Online Article Text |
id | pubmed-10124049 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2023 |
publisher | BioMed Central |
record_format | MEDLINE/PubMed |
spelling | pubmed-101240492023-04-25 Comparison of correctly and incorrectly classified patients for in-hospital mortality prediction in the intensive care unit Stenwig, Eline Salvi, Giampiero Salvo Rossi, Pierluigi Skjærvold, Nils Kristian BMC Med Res Methodol Research BACKGROUND: The use of machine learning is becoming increasingly popular in many disciplines, but there is still an implementation gap of machine learning models in clinical settings. Lack of trust in models is one of the issues that need to be addressed in an effort to close this gap. No models are perfect, and it is crucial to know in which use cases we can trust a model and for which cases it is less reliable. METHODS: Four different algorithms are trained on the eICU Collaborative Research Database using similar features as the APACHE IV severity-of-disease scoring system to predict hospital mortality in the ICU. The training and testing procedure is repeated 100 times on the same dataset to investigate whether predictions for single patients change with small changes in the models. Features are then analysed separately to investigate potential differences between patients consistently classified correctly and incorrectly. RESULTS: A total of 34 056 patients (58.4%) are classified as true negative, 6 527 patients (11.3%) as false positive, 3 984 patients (6.8%) as true positive, and 546 patients (0.9%) as false negatives. The remaining 13 108 patients (22.5%) are inconsistently classified across models and rounds. Histograms and distributions of feature values are compared visually to investigate differences between groups. CONCLUSIONS: It is impossible to distinguish the groups using single features alone. Considering a combination of features, the difference between the groups is clearer. Incorrectly classified patients have features more similar to patients with the same prediction rather than the same outcome. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1186/s12874-023-01921-9. BioMed Central 2023-04-24 /pmc/articles/PMC10124049/ /pubmed/37095430 http://dx.doi.org/10.1186/s12874-023-01921-9 Text en © The Author(s) 2023 https://creativecommons.org/licenses/by/4.0/Open AccessThis article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ (https://creativecommons.org/licenses/by/4.0/) . The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/ (https://creativecommons.org/publicdomain/zero/1.0/) ) applies to the data made available in this article, unless otherwise stated in a credit line to the data. |
spellingShingle | Research Stenwig, Eline Salvi, Giampiero Salvo Rossi, Pierluigi Skjærvold, Nils Kristian Comparison of correctly and incorrectly classified patients for in-hospital mortality prediction in the intensive care unit |
title | Comparison of correctly and incorrectly classified patients for in-hospital mortality prediction in the intensive care unit |
title_full | Comparison of correctly and incorrectly classified patients for in-hospital mortality prediction in the intensive care unit |
title_fullStr | Comparison of correctly and incorrectly classified patients for in-hospital mortality prediction in the intensive care unit |
title_full_unstemmed | Comparison of correctly and incorrectly classified patients for in-hospital mortality prediction in the intensive care unit |
title_short | Comparison of correctly and incorrectly classified patients for in-hospital mortality prediction in the intensive care unit |
title_sort | comparison of correctly and incorrectly classified patients for in-hospital mortality prediction in the intensive care unit |
topic | Research |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10124049/ https://www.ncbi.nlm.nih.gov/pubmed/37095430 http://dx.doi.org/10.1186/s12874-023-01921-9 |
work_keys_str_mv | AT stenwigeline comparisonofcorrectlyandincorrectlyclassifiedpatientsforinhospitalmortalitypredictionintheintensivecareunit AT salvigiampiero comparisonofcorrectlyandincorrectlyclassifiedpatientsforinhospitalmortalitypredictionintheintensivecareunit AT salvorossipierluigi comparisonofcorrectlyandincorrectlyclassifiedpatientsforinhospitalmortalitypredictionintheintensivecareunit AT skjærvoldnilskristian comparisonofcorrectlyandincorrectlyclassifiedpatientsforinhospitalmortalitypredictionintheintensivecareunit |