Cargando…

Almanac: Retrieval-Augmented Language Models for Clinical Medicine

Large-language models have recently demonstrated impressive zero-shot capabilities in a variety of natural language tasks such as summarization, dialogue generation, and question-answering. Despite many promising applications in clinical medicine, adoption of these models in real-world settings has...

Descripción completa

Detalles Bibliográficos
Autores principales: Zakka, Cyril, Chaurasia, Akash, Shad, Rohan, Dalal, Alex R., Kim, Jennifer L., Moor, Michael, Alexander, Kevin, Ashley, Euan, Boyd, Jack, Boyd, Kathleen, Hirsch, Karen, Langlotz, Curt, Nelson, Joanna, Hiesinger, William
Formato: Online Artículo Texto
Lenguaje:English
Publicado: American Journal Experts 2023
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10187428/
https://www.ncbi.nlm.nih.gov/pubmed/37205549
http://dx.doi.org/10.21203/rs.3.rs-2883198/v1
_version_ 1785042735742320640
author Zakka, Cyril
Chaurasia, Akash
Shad, Rohan
Dalal, Alex R.
Kim, Jennifer L.
Moor, Michael
Alexander, Kevin
Ashley, Euan
Boyd, Jack
Boyd, Kathleen
Hirsch, Karen
Langlotz, Curt
Nelson, Joanna
Hiesinger, William
author_facet Zakka, Cyril
Chaurasia, Akash
Shad, Rohan
Dalal, Alex R.
Kim, Jennifer L.
Moor, Michael
Alexander, Kevin
Ashley, Euan
Boyd, Jack
Boyd, Kathleen
Hirsch, Karen
Langlotz, Curt
Nelson, Joanna
Hiesinger, William
author_sort Zakka, Cyril
collection PubMed
description Large-language models have recently demonstrated impressive zero-shot capabilities in a variety of natural language tasks such as summarization, dialogue generation, and question-answering. Despite many promising applications in clinical medicine, adoption of these models in real-world settings has been largely limited by their tendency to generate incorrect and sometimes even toxic statements. In this study, we develop Almanac, a large language model framework augmented with retrieval capabilities for medical guideline and treatment recommendations. Performance on a novel dataset of clinical scenarios (n= 130) evaluated by a panel of 5 board-certified and resident physicians demonstrates significant increases in factuality (mean of 18% at p-value < 0.05) across all specialties, with improvements in completeness and safety. Our results demonstrate the potential for large language models to be effective tools in the clinical decision-making process, while also emphasizing the importance of careful testing and deployment to mitigate their shortcomings.
format Online
Article
Text
id pubmed-10187428
institution National Center for Biotechnology Information
language English
publishDate 2023
publisher American Journal Experts
record_format MEDLINE/PubMed
spelling pubmed-101874282023-05-17 Almanac: Retrieval-Augmented Language Models for Clinical Medicine Zakka, Cyril Chaurasia, Akash Shad, Rohan Dalal, Alex R. Kim, Jennifer L. Moor, Michael Alexander, Kevin Ashley, Euan Boyd, Jack Boyd, Kathleen Hirsch, Karen Langlotz, Curt Nelson, Joanna Hiesinger, William Res Sq Article Large-language models have recently demonstrated impressive zero-shot capabilities in a variety of natural language tasks such as summarization, dialogue generation, and question-answering. Despite many promising applications in clinical medicine, adoption of these models in real-world settings has been largely limited by their tendency to generate incorrect and sometimes even toxic statements. In this study, we develop Almanac, a large language model framework augmented with retrieval capabilities for medical guideline and treatment recommendations. Performance on a novel dataset of clinical scenarios (n= 130) evaluated by a panel of 5 board-certified and resident physicians demonstrates significant increases in factuality (mean of 18% at p-value < 0.05) across all specialties, with improvements in completeness and safety. Our results demonstrate the potential for large language models to be effective tools in the clinical decision-making process, while also emphasizing the importance of careful testing and deployment to mitigate their shortcomings. American Journal Experts 2023-05-02 /pmc/articles/PMC10187428/ /pubmed/37205549 http://dx.doi.org/10.21203/rs.3.rs-2883198/v1 Text en https://creativecommons.org/licenses/by/4.0/This work is licensed under a Creative Commons Attribution 4.0 International License (https://creativecommons.org/licenses/by/4.0/) , which allows reusers to distribute, remix, adapt, and build upon the material in any medium or format, so long as attribution is given to the creator. The license allows for commercial use. https://creativecommons.org/licenses/by/4.0/License: This work is licensed under a Creative Commons Attribution 4.0 International License. Read Full License (https://creativecommons.org/licenses/by/4.0/)
spellingShingle Article
Zakka, Cyril
Chaurasia, Akash
Shad, Rohan
Dalal, Alex R.
Kim, Jennifer L.
Moor, Michael
Alexander, Kevin
Ashley, Euan
Boyd, Jack
Boyd, Kathleen
Hirsch, Karen
Langlotz, Curt
Nelson, Joanna
Hiesinger, William
Almanac: Retrieval-Augmented Language Models for Clinical Medicine
title Almanac: Retrieval-Augmented Language Models for Clinical Medicine
title_full Almanac: Retrieval-Augmented Language Models for Clinical Medicine
title_fullStr Almanac: Retrieval-Augmented Language Models for Clinical Medicine
title_full_unstemmed Almanac: Retrieval-Augmented Language Models for Clinical Medicine
title_short Almanac: Retrieval-Augmented Language Models for Clinical Medicine
title_sort almanac: retrieval-augmented language models for clinical medicine
topic Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10187428/
https://www.ncbi.nlm.nih.gov/pubmed/37205549
http://dx.doi.org/10.21203/rs.3.rs-2883198/v1
work_keys_str_mv AT zakkacyril almanacretrievalaugmentedlanguagemodelsforclinicalmedicine
AT chaurasiaakash almanacretrievalaugmentedlanguagemodelsforclinicalmedicine
AT shadrohan almanacretrievalaugmentedlanguagemodelsforclinicalmedicine
AT dalalalexr almanacretrievalaugmentedlanguagemodelsforclinicalmedicine
AT kimjenniferl almanacretrievalaugmentedlanguagemodelsforclinicalmedicine
AT moormichael almanacretrievalaugmentedlanguagemodelsforclinicalmedicine
AT alexanderkevin almanacretrievalaugmentedlanguagemodelsforclinicalmedicine
AT ashleyeuan almanacretrievalaugmentedlanguagemodelsforclinicalmedicine
AT boydjack almanacretrievalaugmentedlanguagemodelsforclinicalmedicine
AT boydkathleen almanacretrievalaugmentedlanguagemodelsforclinicalmedicine
AT hirschkaren almanacretrievalaugmentedlanguagemodelsforclinicalmedicine
AT langlotzcurt almanacretrievalaugmentedlanguagemodelsforclinicalmedicine
AT nelsonjoanna almanacretrievalaugmentedlanguagemodelsforclinicalmedicine
AT hiesingerwilliam almanacretrievalaugmentedlanguagemodelsforclinicalmedicine