Cargando…
Identifying Patients With Delirium Based on Unstructured Clinical Notes: Observational Study
BACKGROUND: Delirium in hospitalized patients is a syndrome of acute brain dysfunction. Diagnostic (International Classification of Diseases [ICD]) codes are often used in studies using electronic health records (EHRs), but they are inaccurate. OBJECTIVE: We sought to develop a more accurate method...
Autores principales: | , , , , , , , , , , , , , , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
JMIR Publications
2022
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9270709/ https://www.ncbi.nlm.nih.gov/pubmed/35749214 http://dx.doi.org/10.2196/33834 |
_version_ | 1784744525342703616 |
---|---|
author | Ge, Wendong Alabsi, Haitham Jain, Aayushee Ye, Elissa Sun, Haoqi Fernandes, Marta Magdamo, Colin Tesh, Ryan A Collens, Sarah I Newhouse, Amy MVR Moura, Lidia Zafar, Sahar Hsu, John Akeju, Oluwaseun Robbins, Gregory K Mukerji, Shibani S Das, Sudeshna Westover, M Brandon |
author_facet | Ge, Wendong Alabsi, Haitham Jain, Aayushee Ye, Elissa Sun, Haoqi Fernandes, Marta Magdamo, Colin Tesh, Ryan A Collens, Sarah I Newhouse, Amy MVR Moura, Lidia Zafar, Sahar Hsu, John Akeju, Oluwaseun Robbins, Gregory K Mukerji, Shibani S Das, Sudeshna Westover, M Brandon |
author_sort | Ge, Wendong |
collection | PubMed |
description | BACKGROUND: Delirium in hospitalized patients is a syndrome of acute brain dysfunction. Diagnostic (International Classification of Diseases [ICD]) codes are often used in studies using electronic health records (EHRs), but they are inaccurate. OBJECTIVE: We sought to develop a more accurate method using natural language processing (NLP) to detect delirium episodes on the basis of unstructured clinical notes. METHODS: We collected 1.5 million notes from >10,000 patients from among 9 hospitals. Seven experts iteratively labeled 200,471 sentences. Using these, we trained three NLP classifiers: Support Vector Machine, Recurrent Neural Networks, and Transformer. Testing was performed using an external data set. We also evaluated associations with delirium billing (ICD) codes, medications, orders for restraints and sitters, direct assessments (Confusion Assessment Method [CAM] scores), and in-hospital mortality. F1 scores, confusion matrices, and areas under the receiver operating characteristic curve (AUCs) were used to compare NLP models. We used the φ coefficient to measure associations with other delirium indicators. RESULTS: The transformer NLP performed best on the following parameters: micro F1=0.978, macro F1=0.918, positive AUC=0.984, and negative AUC=0.992. NLP detections exhibited higher correlations (φ) than ICD codes with deliriogenic medications (0.194 vs 0.073 for ICD codes), restraints and sitter orders (0.358 vs 0.177), mortality (0.216 vs 0.000), and CAM scores (0.256 vs –0.028). CONCLUSIONS: Clinical notes are an attractive alternative to ICD codes for EHR delirium studies but require automated methods. Our NLP model detects delirium with high accuracy, similar to manual chart review. Our NLP approach can provide more accurate determination of delirium for large-scale EHR-based studies regarding delirium, quality improvement, and clinical trails. |
format | Online Article Text |
id | pubmed-9270709 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2022 |
publisher | JMIR Publications |
record_format | MEDLINE/PubMed |
spelling | pubmed-92707092022-07-10 Identifying Patients With Delirium Based on Unstructured Clinical Notes: Observational Study Ge, Wendong Alabsi, Haitham Jain, Aayushee Ye, Elissa Sun, Haoqi Fernandes, Marta Magdamo, Colin Tesh, Ryan A Collens, Sarah I Newhouse, Amy MVR Moura, Lidia Zafar, Sahar Hsu, John Akeju, Oluwaseun Robbins, Gregory K Mukerji, Shibani S Das, Sudeshna Westover, M Brandon JMIR Form Res Original Paper BACKGROUND: Delirium in hospitalized patients is a syndrome of acute brain dysfunction. Diagnostic (International Classification of Diseases [ICD]) codes are often used in studies using electronic health records (EHRs), but they are inaccurate. OBJECTIVE: We sought to develop a more accurate method using natural language processing (NLP) to detect delirium episodes on the basis of unstructured clinical notes. METHODS: We collected 1.5 million notes from >10,000 patients from among 9 hospitals. Seven experts iteratively labeled 200,471 sentences. Using these, we trained three NLP classifiers: Support Vector Machine, Recurrent Neural Networks, and Transformer. Testing was performed using an external data set. We also evaluated associations with delirium billing (ICD) codes, medications, orders for restraints and sitters, direct assessments (Confusion Assessment Method [CAM] scores), and in-hospital mortality. F1 scores, confusion matrices, and areas under the receiver operating characteristic curve (AUCs) were used to compare NLP models. We used the φ coefficient to measure associations with other delirium indicators. RESULTS: The transformer NLP performed best on the following parameters: micro F1=0.978, macro F1=0.918, positive AUC=0.984, and negative AUC=0.992. NLP detections exhibited higher correlations (φ) than ICD codes with deliriogenic medications (0.194 vs 0.073 for ICD codes), restraints and sitter orders (0.358 vs 0.177), mortality (0.216 vs 0.000), and CAM scores (0.256 vs –0.028). CONCLUSIONS: Clinical notes are an attractive alternative to ICD codes for EHR delirium studies but require automated methods. Our NLP model detects delirium with high accuracy, similar to manual chart review. Our NLP approach can provide more accurate determination of delirium for large-scale EHR-based studies regarding delirium, quality improvement, and clinical trails. JMIR Publications 2022-06-24 /pmc/articles/PMC9270709/ /pubmed/35749214 http://dx.doi.org/10.2196/33834 Text en ©Wendong Ge, Haitham Alabsi, Aayushee Jain, Elissa Ye, Haoqi Sun, Marta Fernandes, Colin Magdamo, Ryan A Tesh, Sarah I Collens, Amy Newhouse, Lidia MVR Moura, Sahar Zafar, John Hsu, Oluwaseun Akeju, Gregory K Robbins, Shibani S Mukerji, Sudeshna Das, M Brandon Westover. Originally published in JMIR Formative Research (https://formative.jmir.org), 24.06.2022. https://creativecommons.org/licenses/by/4.0/This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included. |
spellingShingle | Original Paper Ge, Wendong Alabsi, Haitham Jain, Aayushee Ye, Elissa Sun, Haoqi Fernandes, Marta Magdamo, Colin Tesh, Ryan A Collens, Sarah I Newhouse, Amy MVR Moura, Lidia Zafar, Sahar Hsu, John Akeju, Oluwaseun Robbins, Gregory K Mukerji, Shibani S Das, Sudeshna Westover, M Brandon Identifying Patients With Delirium Based on Unstructured Clinical Notes: Observational Study |
title | Identifying Patients With Delirium Based on Unstructured Clinical Notes: Observational Study |
title_full | Identifying Patients With Delirium Based on Unstructured Clinical Notes: Observational Study |
title_fullStr | Identifying Patients With Delirium Based on Unstructured Clinical Notes: Observational Study |
title_full_unstemmed | Identifying Patients With Delirium Based on Unstructured Clinical Notes: Observational Study |
title_short | Identifying Patients With Delirium Based on Unstructured Clinical Notes: Observational Study |
title_sort | identifying patients with delirium based on unstructured clinical notes: observational study |
topic | Original Paper |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9270709/ https://www.ncbi.nlm.nih.gov/pubmed/35749214 http://dx.doi.org/10.2196/33834 |
work_keys_str_mv | AT gewendong identifyingpatientswithdeliriumbasedonunstructuredclinicalnotesobservationalstudy AT alabsihaitham identifyingpatientswithdeliriumbasedonunstructuredclinicalnotesobservationalstudy AT jainaayushee identifyingpatientswithdeliriumbasedonunstructuredclinicalnotesobservationalstudy AT yeelissa identifyingpatientswithdeliriumbasedonunstructuredclinicalnotesobservationalstudy AT sunhaoqi identifyingpatientswithdeliriumbasedonunstructuredclinicalnotesobservationalstudy AT fernandesmarta identifyingpatientswithdeliriumbasedonunstructuredclinicalnotesobservationalstudy AT magdamocolin identifyingpatientswithdeliriumbasedonunstructuredclinicalnotesobservationalstudy AT teshryana identifyingpatientswithdeliriumbasedonunstructuredclinicalnotesobservationalstudy AT collenssarahi identifyingpatientswithdeliriumbasedonunstructuredclinicalnotesobservationalstudy AT newhouseamy identifyingpatientswithdeliriumbasedonunstructuredclinicalnotesobservationalstudy AT mvrmouralidia identifyingpatientswithdeliriumbasedonunstructuredclinicalnotesobservationalstudy AT zafarsahar identifyingpatientswithdeliriumbasedonunstructuredclinicalnotesobservationalstudy AT hsujohn identifyingpatientswithdeliriumbasedonunstructuredclinicalnotesobservationalstudy AT akejuoluwaseun identifyingpatientswithdeliriumbasedonunstructuredclinicalnotesobservationalstudy AT robbinsgregoryk identifyingpatientswithdeliriumbasedonunstructuredclinicalnotesobservationalstudy AT mukerjishibanis identifyingpatientswithdeliriumbasedonunstructuredclinicalnotesobservationalstudy AT dassudeshna identifyingpatientswithdeliriumbasedonunstructuredclinicalnotesobservationalstudy AT westovermbrandon identifyingpatientswithdeliriumbasedonunstructuredclinicalnotesobservationalstudy |