Cargando…

Machine Learning Approach to Inpatient Violence Risk Assessment Using Routinely Collected Clinical Notes in Electronic Health Records

IMPORTANCE: Inpatient violence remains a significant problem despite existing risk assessment methods. The lack of robustness and the high degree of effort needed to use current methods might be mitigated by using routinely registered clinical notes. OBJECTIVE: To develop and validate a multivariabl...

Descripción completa

Detalles Bibliográficos
Autores principales: Menger, Vincent, Spruit, Marco, van Est, Roel, Nap, Eline, Scheepers, Floor
Formato: Online Artículo Texto
Lenguaje:English
Publicado: American Medical Association 2019
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6613290/
https://www.ncbi.nlm.nih.gov/pubmed/31268542
http://dx.doi.org/10.1001/jamanetworkopen.2019.6709
_version_ 1783433023713181696
author Menger, Vincent
Spruit, Marco
van Est, Roel
Nap, Eline
Scheepers, Floor
author_facet Menger, Vincent
Spruit, Marco
van Est, Roel
Nap, Eline
Scheepers, Floor
author_sort Menger, Vincent
collection PubMed
description IMPORTANCE: Inpatient violence remains a significant problem despite existing risk assessment methods. The lack of robustness and the high degree of effort needed to use current methods might be mitigated by using routinely registered clinical notes. OBJECTIVE: To develop and validate a multivariable prediction model for assessing inpatient violence risk based on machine learning techniques applied to clinical notes written in patients’ electronic health records. DESIGN, SETTING, AND PARTICIPANTS: This prognostic study used retrospective clinical notes registered in electronic health records during admission at 2 independent psychiatric health care institutions in the Netherlands. No exclusion criteria for individual patients were defined. At site 1, all adults admitted between January 2013 and August 2018 were included, and at site 2 all adults admitted to general psychiatric wards between June 2016 and August 2018 were included. Data were analyzed between September 2018 and February 2019. MAIN OUTCOMES AND MEASURES: Predictive validity and generalizability of prognostic models measured using area under the curve (AUC). RESULTS: Clinical notes recorded during a total of 3189 admissions of 2209 unique individuals at site 1 (mean [SD] age, 34.0 [16.6] years; 1536 [48.2%] male) and 3253 admissions of 1919 unique individuals at site 2 (mean [SD] age, 45.9 [16.6] years; 2097 [64.5%] male) were analyzed. Violent outcome was determined using the Staff Observation Aggression Scale–Revised. Nested cross-validation was used to train and evaluate models that assess violence risk during the first 4 weeks of admission based on clinical notes available after 24 hours. The predictive validity of models was measured at site 1 (AUC = 0.797; 95% CI, 0.771-0.822) and site 2 (AUC = 0.764; 95% CI, 0.732-0.797). The validation of pretrained models in the other site resulted in AUCs of 0.722 (95% CI, 0.690-0.753) at site 1 and 0.643 (95% CI, 0.610-0.675) at site 2; the difference in AUCs between the internally trained model and the model trained on other-site data was significant at site 1 (AUC difference = 0.075; 95% CI, 0.045-0.105; P < .001) and site 2 (AUC difference = 0.121; 95% CI, 0.085-0.156; P < .001). CONCLUSIONS AND RELEVANCE: Internally validated predictions resulted in AUC values with good predictive validity, suggesting that automatic violence risk assessment using routinely registered clinical notes is possible. The validation of trained models using data from other sites corroborates previous findings that violence risk assessment generalizes modestly to different populations.
format Online
Article
Text
id pubmed-6613290
institution National Center for Biotechnology Information
language English
publishDate 2019
publisher American Medical Association
record_format MEDLINE/PubMed
spelling pubmed-66132902019-07-23 Machine Learning Approach to Inpatient Violence Risk Assessment Using Routinely Collected Clinical Notes in Electronic Health Records Menger, Vincent Spruit, Marco van Est, Roel Nap, Eline Scheepers, Floor JAMA Netw Open Original Investigation IMPORTANCE: Inpatient violence remains a significant problem despite existing risk assessment methods. The lack of robustness and the high degree of effort needed to use current methods might be mitigated by using routinely registered clinical notes. OBJECTIVE: To develop and validate a multivariable prediction model for assessing inpatient violence risk based on machine learning techniques applied to clinical notes written in patients’ electronic health records. DESIGN, SETTING, AND PARTICIPANTS: This prognostic study used retrospective clinical notes registered in electronic health records during admission at 2 independent psychiatric health care institutions in the Netherlands. No exclusion criteria for individual patients were defined. At site 1, all adults admitted between January 2013 and August 2018 were included, and at site 2 all adults admitted to general psychiatric wards between June 2016 and August 2018 were included. Data were analyzed between September 2018 and February 2019. MAIN OUTCOMES AND MEASURES: Predictive validity and generalizability of prognostic models measured using area under the curve (AUC). RESULTS: Clinical notes recorded during a total of 3189 admissions of 2209 unique individuals at site 1 (mean [SD] age, 34.0 [16.6] years; 1536 [48.2%] male) and 3253 admissions of 1919 unique individuals at site 2 (mean [SD] age, 45.9 [16.6] years; 2097 [64.5%] male) were analyzed. Violent outcome was determined using the Staff Observation Aggression Scale–Revised. Nested cross-validation was used to train and evaluate models that assess violence risk during the first 4 weeks of admission based on clinical notes available after 24 hours. The predictive validity of models was measured at site 1 (AUC = 0.797; 95% CI, 0.771-0.822) and site 2 (AUC = 0.764; 95% CI, 0.732-0.797). The validation of pretrained models in the other site resulted in AUCs of 0.722 (95% CI, 0.690-0.753) at site 1 and 0.643 (95% CI, 0.610-0.675) at site 2; the difference in AUCs between the internally trained model and the model trained on other-site data was significant at site 1 (AUC difference = 0.075; 95% CI, 0.045-0.105; P < .001) and site 2 (AUC difference = 0.121; 95% CI, 0.085-0.156; P < .001). CONCLUSIONS AND RELEVANCE: Internally validated predictions resulted in AUC values with good predictive validity, suggesting that automatic violence risk assessment using routinely registered clinical notes is possible. The validation of trained models using data from other sites corroborates previous findings that violence risk assessment generalizes modestly to different populations. American Medical Association 2019-07-03 /pmc/articles/PMC6613290/ /pubmed/31268542 http://dx.doi.org/10.1001/jamanetworkopen.2019.6709 Text en Copyright 2019 Menger V et al. JAMA Network Open. http://creativecommons.org/licenses/by/4.0/ This is an open access article distributed under the terms of the CC-BY License.
spellingShingle Original Investigation
Menger, Vincent
Spruit, Marco
van Est, Roel
Nap, Eline
Scheepers, Floor
Machine Learning Approach to Inpatient Violence Risk Assessment Using Routinely Collected Clinical Notes in Electronic Health Records
title Machine Learning Approach to Inpatient Violence Risk Assessment Using Routinely Collected Clinical Notes in Electronic Health Records
title_full Machine Learning Approach to Inpatient Violence Risk Assessment Using Routinely Collected Clinical Notes in Electronic Health Records
title_fullStr Machine Learning Approach to Inpatient Violence Risk Assessment Using Routinely Collected Clinical Notes in Electronic Health Records
title_full_unstemmed Machine Learning Approach to Inpatient Violence Risk Assessment Using Routinely Collected Clinical Notes in Electronic Health Records
title_short Machine Learning Approach to Inpatient Violence Risk Assessment Using Routinely Collected Clinical Notes in Electronic Health Records
title_sort machine learning approach to inpatient violence risk assessment using routinely collected clinical notes in electronic health records
topic Original Investigation
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6613290/
https://www.ncbi.nlm.nih.gov/pubmed/31268542
http://dx.doi.org/10.1001/jamanetworkopen.2019.6709
work_keys_str_mv AT mengervincent machinelearningapproachtoinpatientviolenceriskassessmentusingroutinelycollectedclinicalnotesinelectronichealthrecords
AT spruitmarco machinelearningapproachtoinpatientviolenceriskassessmentusingroutinelycollectedclinicalnotesinelectronichealthrecords
AT vanestroel machinelearningapproachtoinpatientviolenceriskassessmentusingroutinelycollectedclinicalnotesinelectronichealthrecords
AT napeline machinelearningapproachtoinpatientviolenceriskassessmentusingroutinelycollectedclinicalnotesinelectronichealthrecords
AT scheepersfloor machinelearningapproachtoinpatientviolenceriskassessmentusingroutinelycollectedclinicalnotesinelectronichealthrecords