Cargando…
Almanac: Retrieval-Augmented Language Models for Clinical Medicine
Large-language models have recently demonstrated impressive zero-shot capabilities in a variety of natural language tasks such as summarization, dialogue generation, and question-answering. Despite many promising applications in clinical medicine, adoption of these models in real-world settings has...
Autores principales: | , , , , , , , , , , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
American Journal Experts
2023
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10187428/ https://www.ncbi.nlm.nih.gov/pubmed/37205549 http://dx.doi.org/10.21203/rs.3.rs-2883198/v1 |
_version_ | 1785042735742320640 |
---|---|
author | Zakka, Cyril Chaurasia, Akash Shad, Rohan Dalal, Alex R. Kim, Jennifer L. Moor, Michael Alexander, Kevin Ashley, Euan Boyd, Jack Boyd, Kathleen Hirsch, Karen Langlotz, Curt Nelson, Joanna Hiesinger, William |
author_facet | Zakka, Cyril Chaurasia, Akash Shad, Rohan Dalal, Alex R. Kim, Jennifer L. Moor, Michael Alexander, Kevin Ashley, Euan Boyd, Jack Boyd, Kathleen Hirsch, Karen Langlotz, Curt Nelson, Joanna Hiesinger, William |
author_sort | Zakka, Cyril |
collection | PubMed |
description | Large-language models have recently demonstrated impressive zero-shot capabilities in a variety of natural language tasks such as summarization, dialogue generation, and question-answering. Despite many promising applications in clinical medicine, adoption of these models in real-world settings has been largely limited by their tendency to generate incorrect and sometimes even toxic statements. In this study, we develop Almanac, a large language model framework augmented with retrieval capabilities for medical guideline and treatment recommendations. Performance on a novel dataset of clinical scenarios (n= 130) evaluated by a panel of 5 board-certified and resident physicians demonstrates significant increases in factuality (mean of 18% at p-value < 0.05) across all specialties, with improvements in completeness and safety. Our results demonstrate the potential for large language models to be effective tools in the clinical decision-making process, while also emphasizing the importance of careful testing and deployment to mitigate their shortcomings. |
format | Online Article Text |
id | pubmed-10187428 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2023 |
publisher | American Journal Experts |
record_format | MEDLINE/PubMed |
spelling | pubmed-101874282023-05-17 Almanac: Retrieval-Augmented Language Models for Clinical Medicine Zakka, Cyril Chaurasia, Akash Shad, Rohan Dalal, Alex R. Kim, Jennifer L. Moor, Michael Alexander, Kevin Ashley, Euan Boyd, Jack Boyd, Kathleen Hirsch, Karen Langlotz, Curt Nelson, Joanna Hiesinger, William Res Sq Article Large-language models have recently demonstrated impressive zero-shot capabilities in a variety of natural language tasks such as summarization, dialogue generation, and question-answering. Despite many promising applications in clinical medicine, adoption of these models in real-world settings has been largely limited by their tendency to generate incorrect and sometimes even toxic statements. In this study, we develop Almanac, a large language model framework augmented with retrieval capabilities for medical guideline and treatment recommendations. Performance on a novel dataset of clinical scenarios (n= 130) evaluated by a panel of 5 board-certified and resident physicians demonstrates significant increases in factuality (mean of 18% at p-value < 0.05) across all specialties, with improvements in completeness and safety. Our results demonstrate the potential for large language models to be effective tools in the clinical decision-making process, while also emphasizing the importance of careful testing and deployment to mitigate their shortcomings. American Journal Experts 2023-05-02 /pmc/articles/PMC10187428/ /pubmed/37205549 http://dx.doi.org/10.21203/rs.3.rs-2883198/v1 Text en https://creativecommons.org/licenses/by/4.0/This work is licensed under a Creative Commons Attribution 4.0 International License (https://creativecommons.org/licenses/by/4.0/) , which allows reusers to distribute, remix, adapt, and build upon the material in any medium or format, so long as attribution is given to the creator. The license allows for commercial use. https://creativecommons.org/licenses/by/4.0/License: This work is licensed under a Creative Commons Attribution 4.0 International License. Read Full License (https://creativecommons.org/licenses/by/4.0/) |
spellingShingle | Article Zakka, Cyril Chaurasia, Akash Shad, Rohan Dalal, Alex R. Kim, Jennifer L. Moor, Michael Alexander, Kevin Ashley, Euan Boyd, Jack Boyd, Kathleen Hirsch, Karen Langlotz, Curt Nelson, Joanna Hiesinger, William Almanac: Retrieval-Augmented Language Models for Clinical Medicine |
title | Almanac: Retrieval-Augmented Language Models for Clinical Medicine |
title_full | Almanac: Retrieval-Augmented Language Models for Clinical Medicine |
title_fullStr | Almanac: Retrieval-Augmented Language Models for Clinical Medicine |
title_full_unstemmed | Almanac: Retrieval-Augmented Language Models for Clinical Medicine |
title_short | Almanac: Retrieval-Augmented Language Models for Clinical Medicine |
title_sort | almanac: retrieval-augmented language models for clinical medicine |
topic | Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10187428/ https://www.ncbi.nlm.nih.gov/pubmed/37205549 http://dx.doi.org/10.21203/rs.3.rs-2883198/v1 |
work_keys_str_mv | AT zakkacyril almanacretrievalaugmentedlanguagemodelsforclinicalmedicine AT chaurasiaakash almanacretrievalaugmentedlanguagemodelsforclinicalmedicine AT shadrohan almanacretrievalaugmentedlanguagemodelsforclinicalmedicine AT dalalalexr almanacretrievalaugmentedlanguagemodelsforclinicalmedicine AT kimjenniferl almanacretrievalaugmentedlanguagemodelsforclinicalmedicine AT moormichael almanacretrievalaugmentedlanguagemodelsforclinicalmedicine AT alexanderkevin almanacretrievalaugmentedlanguagemodelsforclinicalmedicine AT ashleyeuan almanacretrievalaugmentedlanguagemodelsforclinicalmedicine AT boydjack almanacretrievalaugmentedlanguagemodelsforclinicalmedicine AT boydkathleen almanacretrievalaugmentedlanguagemodelsforclinicalmedicine AT hirschkaren almanacretrievalaugmentedlanguagemodelsforclinicalmedicine AT langlotzcurt almanacretrievalaugmentedlanguagemodelsforclinicalmedicine AT nelsonjoanna almanacretrievalaugmentedlanguagemodelsforclinicalmedicine AT hiesingerwilliam almanacretrievalaugmentedlanguagemodelsforclinicalmedicine |