Cargando…

Classifying Firearm Injury Intent in Electronic Hospital Records Using Natural Language Processing

IMPORTANCE: International Classification of Diseases–coded hospital discharge data do not accurately reflect whether firearm injuries were caused by assault, unintentional injury, self-harm, legal intervention, or were of undetermined intent. Applying natural language processing (NLP) and machine le...

Descripción completa

Detalles Bibliográficos
Autores principales: MacPhaul, Erin, Zhou, Li, Mooney, Stephen J., Azrael, Deborah, Bowen, Andrew, Rowhani-Rahbar, Ali, Yenduri, Ravali, Barber, Catherine, Goralnick, Eric, Miller, Matthew
Formato: Online Artículo Texto
Lenguaje:English
Publicado: American Medical Association 2023
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10080369/
https://www.ncbi.nlm.nih.gov/pubmed/37022685
http://dx.doi.org/10.1001/jamanetworkopen.2023.5870
_version_ 1785020908433309696
author MacPhaul, Erin
Zhou, Li
Mooney, Stephen J.
Azrael, Deborah
Bowen, Andrew
Rowhani-Rahbar, Ali
Yenduri, Ravali
Barber, Catherine
Goralnick, Eric
Miller, Matthew
author_facet MacPhaul, Erin
Zhou, Li
Mooney, Stephen J.
Azrael, Deborah
Bowen, Andrew
Rowhani-Rahbar, Ali
Yenduri, Ravali
Barber, Catherine
Goralnick, Eric
Miller, Matthew
author_sort MacPhaul, Erin
collection PubMed
description IMPORTANCE: International Classification of Diseases–coded hospital discharge data do not accurately reflect whether firearm injuries were caused by assault, unintentional injury, self-harm, legal intervention, or were of undetermined intent. Applying natural language processing (NLP) and machine learning (ML) techniques to electronic health record (EHR) narrative text could be associated with improved accuracy of firearm injury intent data. OBJECTIVE: To assess the accuracy with which an ML model identified firearm injury intent. DESIGN, SETTING, AND PARTICIPANTS: A cross-sectional retrospective EHR review was conducted at 3 level I trauma centers, 2 from health care institutions in Boston, Massachusetts, and 1 from Seattle, Washington, between January 1, 2000, and December 31, 2019; data analysis was performed from January 18, 2021, to August 22, 2022. A total of 1915 incident cases of firearm injury in patients presenting to emergency departments at the model development institution and 769 from the external validation institution with a firearm injury code assigned according to International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) or International Statistical Classification of Diseases and Related Health Problems, 10th Revision, Clinical Modification (ICD-10-CM), in discharge data were included. EXPOSURES: Classification of firearm injury intent. MAIN OUTCOMES AND MEASURES: Intent classification accuracy by the NLP model was compared with ICD codes assigned by medical record coders in discharge data. The NLP model extracted intent-relevant features from narrative text that were then used by a gradient-boosting classifier to determine the intent of each firearm injury. Classification accuracy was evaluated against intent assigned by the research team. The model was further validated using an external data set. RESULTS: The NLP model was evaluated in 381 patients presenting with firearm injury at the model development site (mean [SD] age, 39.2 [13.0] years; 348 [91.3%] men) and 304 patients at the external development site (mean [SD] age, 31.8 [14.8] years; 263 [86.5%] men). The model proved more accurate than medical record coders in assigning intent to firearm injuries at the model development site (accident F-score, 0.78 vs 0.40; assault F-score, 0.90 vs 0.78). The model maintained this improvement on an external validation set from a second institution (accident F-score, 0.64 vs 0.58; assault F-score, 0.88 vs 0.81). While the model showed some degradation between institutions, retraining the model using data from the second institution further improved performance on that site’s records (accident F-score, 0.75; assault F-score, 0.92). CONCLUSIONS AND RELEVANCE: The findings of this study suggest that NLP ML can be used to improve the accuracy of firearm injury intent classification compared with ICD-coded discharge data, particularly for cases of accident and assault intents (the most prevalent and commonly misclassified intent types). Future research could refine this model using larger and more diverse data sets.
format Online
Article
Text
id pubmed-10080369
institution National Center for Biotechnology Information
language English
publishDate 2023
publisher American Medical Association
record_format MEDLINE/PubMed
spelling pubmed-100803692023-04-08 Classifying Firearm Injury Intent in Electronic Hospital Records Using Natural Language Processing MacPhaul, Erin Zhou, Li Mooney, Stephen J. Azrael, Deborah Bowen, Andrew Rowhani-Rahbar, Ali Yenduri, Ravali Barber, Catherine Goralnick, Eric Miller, Matthew JAMA Netw Open Original Investigation IMPORTANCE: International Classification of Diseases–coded hospital discharge data do not accurately reflect whether firearm injuries were caused by assault, unintentional injury, self-harm, legal intervention, or were of undetermined intent. Applying natural language processing (NLP) and machine learning (ML) techniques to electronic health record (EHR) narrative text could be associated with improved accuracy of firearm injury intent data. OBJECTIVE: To assess the accuracy with which an ML model identified firearm injury intent. DESIGN, SETTING, AND PARTICIPANTS: A cross-sectional retrospective EHR review was conducted at 3 level I trauma centers, 2 from health care institutions in Boston, Massachusetts, and 1 from Seattle, Washington, between January 1, 2000, and December 31, 2019; data analysis was performed from January 18, 2021, to August 22, 2022. A total of 1915 incident cases of firearm injury in patients presenting to emergency departments at the model development institution and 769 from the external validation institution with a firearm injury code assigned according to International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) or International Statistical Classification of Diseases and Related Health Problems, 10th Revision, Clinical Modification (ICD-10-CM), in discharge data were included. EXPOSURES: Classification of firearm injury intent. MAIN OUTCOMES AND MEASURES: Intent classification accuracy by the NLP model was compared with ICD codes assigned by medical record coders in discharge data. The NLP model extracted intent-relevant features from narrative text that were then used by a gradient-boosting classifier to determine the intent of each firearm injury. Classification accuracy was evaluated against intent assigned by the research team. The model was further validated using an external data set. RESULTS: The NLP model was evaluated in 381 patients presenting with firearm injury at the model development site (mean [SD] age, 39.2 [13.0] years; 348 [91.3%] men) and 304 patients at the external development site (mean [SD] age, 31.8 [14.8] years; 263 [86.5%] men). The model proved more accurate than medical record coders in assigning intent to firearm injuries at the model development site (accident F-score, 0.78 vs 0.40; assault F-score, 0.90 vs 0.78). The model maintained this improvement on an external validation set from a second institution (accident F-score, 0.64 vs 0.58; assault F-score, 0.88 vs 0.81). While the model showed some degradation between institutions, retraining the model using data from the second institution further improved performance on that site’s records (accident F-score, 0.75; assault F-score, 0.92). CONCLUSIONS AND RELEVANCE: The findings of this study suggest that NLP ML can be used to improve the accuracy of firearm injury intent classification compared with ICD-coded discharge data, particularly for cases of accident and assault intents (the most prevalent and commonly misclassified intent types). Future research could refine this model using larger and more diverse data sets. American Medical Association 2023-04-06 /pmc/articles/PMC10080369/ /pubmed/37022685 http://dx.doi.org/10.1001/jamanetworkopen.2023.5870 Text en Copyright 2023 MacPhaul E et al. JAMA Network Open. https://creativecommons.org/licenses/by/4.0/This is an open access article distributed under the terms of the CC-BY License.
spellingShingle Original Investigation
MacPhaul, Erin
Zhou, Li
Mooney, Stephen J.
Azrael, Deborah
Bowen, Andrew
Rowhani-Rahbar, Ali
Yenduri, Ravali
Barber, Catherine
Goralnick, Eric
Miller, Matthew
Classifying Firearm Injury Intent in Electronic Hospital Records Using Natural Language Processing
title Classifying Firearm Injury Intent in Electronic Hospital Records Using Natural Language Processing
title_full Classifying Firearm Injury Intent in Electronic Hospital Records Using Natural Language Processing
title_fullStr Classifying Firearm Injury Intent in Electronic Hospital Records Using Natural Language Processing
title_full_unstemmed Classifying Firearm Injury Intent in Electronic Hospital Records Using Natural Language Processing
title_short Classifying Firearm Injury Intent in Electronic Hospital Records Using Natural Language Processing
title_sort classifying firearm injury intent in electronic hospital records using natural language processing
topic Original Investigation
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10080369/
https://www.ncbi.nlm.nih.gov/pubmed/37022685
http://dx.doi.org/10.1001/jamanetworkopen.2023.5870
work_keys_str_mv AT macphaulerin classifyingfirearminjuryintentinelectronichospitalrecordsusingnaturallanguageprocessing
AT zhouli classifyingfirearminjuryintentinelectronichospitalrecordsusingnaturallanguageprocessing
AT mooneystephenj classifyingfirearminjuryintentinelectronichospitalrecordsusingnaturallanguageprocessing
AT azraeldeborah classifyingfirearminjuryintentinelectronichospitalrecordsusingnaturallanguageprocessing
AT bowenandrew classifyingfirearminjuryintentinelectronichospitalrecordsusingnaturallanguageprocessing
AT rowhanirahbarali classifyingfirearminjuryintentinelectronichospitalrecordsusingnaturallanguageprocessing
AT yenduriravali classifyingfirearminjuryintentinelectronichospitalrecordsusingnaturallanguageprocessing
AT barbercatherine classifyingfirearminjuryintentinelectronichospitalrecordsusingnaturallanguageprocessing
AT goralnickeric classifyingfirearminjuryintentinelectronichospitalrecordsusingnaturallanguageprocessing
AT millermatthew classifyingfirearminjuryintentinelectronichospitalrecordsusingnaturallanguageprocessing