Cargando…

Natural Language Processing for Adjudication of Heart Failure Hospitalizations in a Multi-Center Clinical Trial

BACKGROUND: The gold standard for outcome adjudication in clinical trials is chart review by a physician clinical events committee (CEC), which requires substantial time and expertise. Automated adjudication by natural language processing (NLP) may offer a more resource-efficient alternative. We pre...

Descripción completa

Detalles Bibliográficos
Autores principales: Cunningham, Jonathan W., Singh, Pulkit, Reeder, Christopher, Claggett, Brian, Marti-Castellote, Pablo M., Lau, Emily S., Khurshid, Shaan, Batra, Puneet, Lubitz, Steven A., Maddah, Mahnaz, Philippakis, Anthony, Desai, Akshay S., Ellinor, Patrick T., Vardeny, Orly, Solomon, Scott D., Ho, Jennifer E.
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Cold Spring Harbor Laboratory 2023
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10473787/
https://www.ncbi.nlm.nih.gov/pubmed/37662283
http://dx.doi.org/10.1101/2023.08.17.23294234
_version_ 1785100336032120832
author Cunningham, Jonathan W.
Singh, Pulkit
Reeder, Christopher
Claggett, Brian
Marti-Castellote, Pablo M.
Lau, Emily S.
Khurshid, Shaan
Batra, Puneet
Lubitz, Steven A.
Maddah, Mahnaz
Philippakis, Anthony
Desai, Akshay S.
Ellinor, Patrick T.
Vardeny, Orly
Solomon, Scott D.
Ho, Jennifer E.
author_facet Cunningham, Jonathan W.
Singh, Pulkit
Reeder, Christopher
Claggett, Brian
Marti-Castellote, Pablo M.
Lau, Emily S.
Khurshid, Shaan
Batra, Puneet
Lubitz, Steven A.
Maddah, Mahnaz
Philippakis, Anthony
Desai, Akshay S.
Ellinor, Patrick T.
Vardeny, Orly
Solomon, Scott D.
Ho, Jennifer E.
author_sort Cunningham, Jonathan W.
collection PubMed
description BACKGROUND: The gold standard for outcome adjudication in clinical trials is chart review by a physician clinical events committee (CEC), which requires substantial time and expertise. Automated adjudication by natural language processing (NLP) may offer a more resource-efficient alternative. We previously showed that the Community Care Cohort Project (C3PO) NLP model adjudicates heart failure (HF) hospitalizations accurately within one healthcare system. METHODS: This study externally validated the C3PO NLP model against CEC adjudication in the INVESTED trial. INVESTED compared influenza vaccination formulations in 5260 patients with cardiovascular disease at 157 North American sites. A central CEC adjudicated the cause of hospitalizations from medical records. We applied the C3PO NLP model to medical records from 4060 INVESTED hospitalizations and evaluated agreement between the NLP and final consensus CEC HF adjudications. We then fine-tuned the C3PO NLP model (C3PO+INVESTED) and trained a de novo model using half the INVESTED hospitalizations, and evaluated these models in the other half. NLP performance was benchmarked to CEC reviewer inter-rater reproducibility. RESULTS: 1074 hospitalizations (26%) were adjudicated as HF by the CEC. There was high agreement between the C3PO NLP and CEC HF adjudications (agreement 87%, kappa statistic 0.69). C3PO NLP model sensitivity was 94% and specificity was 84%. The fine-tuned C3PO and de novo NLP models demonstrated agreement of 93% and kappa of 0.82 and 0.83, respectively. CEC reviewer inter-rater reproducibility was 94% (kappa 0.85). CONCLUSION: Our NLP model developed within a single healthcare system accurately identified HF events relative to the gold-standard CEC in an external multi-center clinical trial. Fine-tuning the model improved agreement and approximated human reproducibility. NLP may improve the efficiency of future multi-center clinical trials by accurately identifying clinical events at scale.
format Online
Article
Text
id pubmed-10473787
institution National Center for Biotechnology Information
language English
publishDate 2023
publisher Cold Spring Harbor Laboratory
record_format MEDLINE/PubMed
spelling pubmed-104737872023-09-02 Natural Language Processing for Adjudication of Heart Failure Hospitalizations in a Multi-Center Clinical Trial Cunningham, Jonathan W. Singh, Pulkit Reeder, Christopher Claggett, Brian Marti-Castellote, Pablo M. Lau, Emily S. Khurshid, Shaan Batra, Puneet Lubitz, Steven A. Maddah, Mahnaz Philippakis, Anthony Desai, Akshay S. Ellinor, Patrick T. Vardeny, Orly Solomon, Scott D. Ho, Jennifer E. medRxiv Article BACKGROUND: The gold standard for outcome adjudication in clinical trials is chart review by a physician clinical events committee (CEC), which requires substantial time and expertise. Automated adjudication by natural language processing (NLP) may offer a more resource-efficient alternative. We previously showed that the Community Care Cohort Project (C3PO) NLP model adjudicates heart failure (HF) hospitalizations accurately within one healthcare system. METHODS: This study externally validated the C3PO NLP model against CEC adjudication in the INVESTED trial. INVESTED compared influenza vaccination formulations in 5260 patients with cardiovascular disease at 157 North American sites. A central CEC adjudicated the cause of hospitalizations from medical records. We applied the C3PO NLP model to medical records from 4060 INVESTED hospitalizations and evaluated agreement between the NLP and final consensus CEC HF adjudications. We then fine-tuned the C3PO NLP model (C3PO+INVESTED) and trained a de novo model using half the INVESTED hospitalizations, and evaluated these models in the other half. NLP performance was benchmarked to CEC reviewer inter-rater reproducibility. RESULTS: 1074 hospitalizations (26%) were adjudicated as HF by the CEC. There was high agreement between the C3PO NLP and CEC HF adjudications (agreement 87%, kappa statistic 0.69). C3PO NLP model sensitivity was 94% and specificity was 84%. The fine-tuned C3PO and de novo NLP models demonstrated agreement of 93% and kappa of 0.82 and 0.83, respectively. CEC reviewer inter-rater reproducibility was 94% (kappa 0.85). CONCLUSION: Our NLP model developed within a single healthcare system accurately identified HF events relative to the gold-standard CEC in an external multi-center clinical trial. Fine-tuning the model improved agreement and approximated human reproducibility. NLP may improve the efficiency of future multi-center clinical trials by accurately identifying clinical events at scale. Cold Spring Harbor Laboratory 2023-08-23 /pmc/articles/PMC10473787/ /pubmed/37662283 http://dx.doi.org/10.1101/2023.08.17.23294234 Text en https://creativecommons.org/licenses/by-nc-nd/4.0/This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License (https://creativecommons.org/licenses/by-nc-nd/4.0/) , which allows reusers to copy and distribute the material in any medium or format in unadapted form only, for noncommercial purposes only, and only so long as attribution is given to the creator.
spellingShingle Article
Cunningham, Jonathan W.
Singh, Pulkit
Reeder, Christopher
Claggett, Brian
Marti-Castellote, Pablo M.
Lau, Emily S.
Khurshid, Shaan
Batra, Puneet
Lubitz, Steven A.
Maddah, Mahnaz
Philippakis, Anthony
Desai, Akshay S.
Ellinor, Patrick T.
Vardeny, Orly
Solomon, Scott D.
Ho, Jennifer E.
Natural Language Processing for Adjudication of Heart Failure Hospitalizations in a Multi-Center Clinical Trial
title Natural Language Processing for Adjudication of Heart Failure Hospitalizations in a Multi-Center Clinical Trial
title_full Natural Language Processing for Adjudication of Heart Failure Hospitalizations in a Multi-Center Clinical Trial
title_fullStr Natural Language Processing for Adjudication of Heart Failure Hospitalizations in a Multi-Center Clinical Trial
title_full_unstemmed Natural Language Processing for Adjudication of Heart Failure Hospitalizations in a Multi-Center Clinical Trial
title_short Natural Language Processing for Adjudication of Heart Failure Hospitalizations in a Multi-Center Clinical Trial
title_sort natural language processing for adjudication of heart failure hospitalizations in a multi-center clinical trial
topic Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10473787/
https://www.ncbi.nlm.nih.gov/pubmed/37662283
http://dx.doi.org/10.1101/2023.08.17.23294234
work_keys_str_mv AT cunninghamjonathanw naturallanguageprocessingforadjudicationofheartfailurehospitalizationsinamulticenterclinicaltrial
AT singhpulkit naturallanguageprocessingforadjudicationofheartfailurehospitalizationsinamulticenterclinicaltrial
AT reederchristopher naturallanguageprocessingforadjudicationofheartfailurehospitalizationsinamulticenterclinicaltrial
AT claggettbrian naturallanguageprocessingforadjudicationofheartfailurehospitalizationsinamulticenterclinicaltrial
AT marticastellotepablom naturallanguageprocessingforadjudicationofheartfailurehospitalizationsinamulticenterclinicaltrial
AT lauemilys naturallanguageprocessingforadjudicationofheartfailurehospitalizationsinamulticenterclinicaltrial
AT khurshidshaan naturallanguageprocessingforadjudicationofheartfailurehospitalizationsinamulticenterclinicaltrial
AT batrapuneet naturallanguageprocessingforadjudicationofheartfailurehospitalizationsinamulticenterclinicaltrial
AT lubitzstevena naturallanguageprocessingforadjudicationofheartfailurehospitalizationsinamulticenterclinicaltrial
AT maddahmahnaz naturallanguageprocessingforadjudicationofheartfailurehospitalizationsinamulticenterclinicaltrial
AT philippakisanthony naturallanguageprocessingforadjudicationofheartfailurehospitalizationsinamulticenterclinicaltrial
AT desaiakshays naturallanguageprocessingforadjudicationofheartfailurehospitalizationsinamulticenterclinicaltrial
AT ellinorpatrickt naturallanguageprocessingforadjudicationofheartfailurehospitalizationsinamulticenterclinicaltrial
AT vardenyorly naturallanguageprocessingforadjudicationofheartfailurehospitalizationsinamulticenterclinicaltrial
AT solomonscottd naturallanguageprocessingforadjudicationofheartfailurehospitalizationsinamulticenterclinicaltrial
AT hojennifere naturallanguageprocessingforadjudicationofheartfailurehospitalizationsinamulticenterclinicaltrial