Cargando…

Challenging ChatGPT 3.5 in Senology—An Assessment of Concordance with Breast Cancer Tumor Board Decision Making

With the recent diffusion of access to publicly available large language models (LLMs), common interest in generative artificial-intelligence-based applications for medical purposes has skyrocketed. The increased use of these models by tech-savvy patients for personal health issues calls for a scien...

Descripción completa

Detalles Bibliográficos
Autores principales: Griewing, Sebastian, Gremke, Niklas, Wagner, Uwe, Lingenfelder, Michael, Kuhn, Sebastian, Boekhoff, Jelena
Formato: Online Artículo Texto
Lenguaje:English
Publicado: MDPI 2023
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10608120/
https://www.ncbi.nlm.nih.gov/pubmed/37888113
http://dx.doi.org/10.3390/jpm13101502
_version_ 1785127704917442560
author Griewing, Sebastian
Gremke, Niklas
Wagner, Uwe
Lingenfelder, Michael
Kuhn, Sebastian
Boekhoff, Jelena
author_facet Griewing, Sebastian
Gremke, Niklas
Wagner, Uwe
Lingenfelder, Michael
Kuhn, Sebastian
Boekhoff, Jelena
author_sort Griewing, Sebastian
collection PubMed
description With the recent diffusion of access to publicly available large language models (LLMs), common interest in generative artificial-intelligence-based applications for medical purposes has skyrocketed. The increased use of these models by tech-savvy patients for personal health issues calls for a scientific evaluation of whether LLMs provide a satisfactory level of accuracy for treatment decisions. This observational study compares the concordance of treatment recommendations from the popular LLM ChatGPT 3.5 with those of a multidisciplinary tumor board for breast cancer (MTB). The study design builds on previous findings by combining an extended input model with patient profiles reflecting patho- and immunomorphological diversity of primary breast cancer, including primary metastasis and precancerous tumor stages. Overall concordance between the LLM and MTB is reached for half of the patient profiles, including precancerous lesions. In the assessment of invasive breast cancer profiles, the concordance amounts to 58.8%. Nevertheless, as the LLM makes considerably fraudulent decisions at times, we do not identify the current development status of publicly available LLMs to be adequate as a support tool for tumor boards. Gynecological oncologists should familiarize themselves with the capabilities of LLMs in order to understand and utilize their potential while keeping in mind potential risks and limitations.
format Online
Article
Text
id pubmed-10608120
institution National Center for Biotechnology Information
language English
publishDate 2023
publisher MDPI
record_format MEDLINE/PubMed
spelling pubmed-106081202023-10-28 Challenging ChatGPT 3.5 in Senology—An Assessment of Concordance with Breast Cancer Tumor Board Decision Making Griewing, Sebastian Gremke, Niklas Wagner, Uwe Lingenfelder, Michael Kuhn, Sebastian Boekhoff, Jelena J Pers Med Article With the recent diffusion of access to publicly available large language models (LLMs), common interest in generative artificial-intelligence-based applications for medical purposes has skyrocketed. The increased use of these models by tech-savvy patients for personal health issues calls for a scientific evaluation of whether LLMs provide a satisfactory level of accuracy for treatment decisions. This observational study compares the concordance of treatment recommendations from the popular LLM ChatGPT 3.5 with those of a multidisciplinary tumor board for breast cancer (MTB). The study design builds on previous findings by combining an extended input model with patient profiles reflecting patho- and immunomorphological diversity of primary breast cancer, including primary metastasis and precancerous tumor stages. Overall concordance between the LLM and MTB is reached for half of the patient profiles, including precancerous lesions. In the assessment of invasive breast cancer profiles, the concordance amounts to 58.8%. Nevertheless, as the LLM makes considerably fraudulent decisions at times, we do not identify the current development status of publicly available LLMs to be adequate as a support tool for tumor boards. Gynecological oncologists should familiarize themselves with the capabilities of LLMs in order to understand and utilize their potential while keeping in mind potential risks and limitations. MDPI 2023-10-16 /pmc/articles/PMC10608120/ /pubmed/37888113 http://dx.doi.org/10.3390/jpm13101502 Text en © 2023 by the authors. https://creativecommons.org/licenses/by/4.0/Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
spellingShingle Article
Griewing, Sebastian
Gremke, Niklas
Wagner, Uwe
Lingenfelder, Michael
Kuhn, Sebastian
Boekhoff, Jelena
Challenging ChatGPT 3.5 in Senology—An Assessment of Concordance with Breast Cancer Tumor Board Decision Making
title Challenging ChatGPT 3.5 in Senology—An Assessment of Concordance with Breast Cancer Tumor Board Decision Making
title_full Challenging ChatGPT 3.5 in Senology—An Assessment of Concordance with Breast Cancer Tumor Board Decision Making
title_fullStr Challenging ChatGPT 3.5 in Senology—An Assessment of Concordance with Breast Cancer Tumor Board Decision Making
title_full_unstemmed Challenging ChatGPT 3.5 in Senology—An Assessment of Concordance with Breast Cancer Tumor Board Decision Making
title_short Challenging ChatGPT 3.5 in Senology—An Assessment of Concordance with Breast Cancer Tumor Board Decision Making
title_sort challenging chatgpt 3.5 in senology—an assessment of concordance with breast cancer tumor board decision making
topic Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10608120/
https://www.ncbi.nlm.nih.gov/pubmed/37888113
http://dx.doi.org/10.3390/jpm13101502
work_keys_str_mv AT griewingsebastian challengingchatgpt35insenologyanassessmentofconcordancewithbreastcancertumorboarddecisionmaking
AT gremkeniklas challengingchatgpt35insenologyanassessmentofconcordancewithbreastcancertumorboarddecisionmaking
AT wagneruwe challengingchatgpt35insenologyanassessmentofconcordancewithbreastcancertumorboarddecisionmaking
AT lingenfeldermichael challengingchatgpt35insenologyanassessmentofconcordancewithbreastcancertumorboarddecisionmaking
AT kuhnsebastian challengingchatgpt35insenologyanassessmentofconcordancewithbreastcancertumorboarddecisionmaking
AT boekhoffjelena challengingchatgpt35insenologyanassessmentofconcordancewithbreastcancertumorboarddecisionmaking