Cargando…

ChatGPT- versus human-generated answers to frequently asked questions about diabetes: A Turing test-inspired survey among employees of a Danish diabetes center

Large language models have received enormous attention recently with some studies demonstrating their potential clinical value, despite not being trained specifically for this domain. We aimed to investigate whether ChatGPT, a language model optimized for dialogue, can answer frequently asked questi...

Descripción completa

Detalles Bibliográficos
Autores principales: Hulman, Adam, Dollerup, Ole Lindgård, Mortensen, Jesper Friis, Fenech, Matthew E., Norman, Kasper, Støvring, Henrik, Hansen, Troels Krarup
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Public Library of Science 2023
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10470899/
https://www.ncbi.nlm.nih.gov/pubmed/37651381
http://dx.doi.org/10.1371/journal.pone.0290773
_version_ 1785099784710782976
author Hulman, Adam
Dollerup, Ole Lindgård
Mortensen, Jesper Friis
Fenech, Matthew E.
Norman, Kasper
Støvring, Henrik
Hansen, Troels Krarup
author_facet Hulman, Adam
Dollerup, Ole Lindgård
Mortensen, Jesper Friis
Fenech, Matthew E.
Norman, Kasper
Støvring, Henrik
Hansen, Troels Krarup
author_sort Hulman, Adam
collection PubMed
description Large language models have received enormous attention recently with some studies demonstrating their potential clinical value, despite not being trained specifically for this domain. We aimed to investigate whether ChatGPT, a language model optimized for dialogue, can answer frequently asked questions about diabetes. We conducted a closed e-survey among employees of a large Danish diabetes center. The study design was inspired by the Turing test and non-inferiority trials. Our survey included ten questions with two answers each. One of these was written by a human expert, while the other was generated by ChatGPT. Participants had the task to identify the ChatGPT-generated answer. Data was analyzed at the question-level using logistic regression with robust variance estimation with clustering at participant level. In secondary analyses, we investigated the effect of participant characteristics on the outcome. A 55% non-inferiority margin was pre-defined based on precision simulations and had been published as part of the study protocol before data collection began. Among 311 invited individuals, 183 participated in the survey (59% response rate). 64% had heard of ChatGPT before, and 19% had tried it. Overall, participants could identify ChatGPT-generated answers 59.5% (95% CI: 57.0, 62.0) of the time, which was outside of the non-inferiority zone. Among participant characteristics, previous ChatGPT use had the strongest association with the outcome (odds ratio: 1.52 (1.16, 2.00), p = 0.003). Previous users answered 67.4% (61.7, 72.7) of the questions correctly, versus non-users’ 57.6% (54.9, 60.3). Participants could distinguish between ChatGPT-generated and human-written answers somewhat better than flipping a fair coin, which was against our initial hypothesis. Rigorously planned studies are needed to elucidate the risks and benefits of integrating such technologies in routine clinical practice.
format Online
Article
Text
id pubmed-10470899
institution National Center for Biotechnology Information
language English
publishDate 2023
publisher Public Library of Science
record_format MEDLINE/PubMed
spelling pubmed-104708992023-09-01 ChatGPT- versus human-generated answers to frequently asked questions about diabetes: A Turing test-inspired survey among employees of a Danish diabetes center Hulman, Adam Dollerup, Ole Lindgård Mortensen, Jesper Friis Fenech, Matthew E. Norman, Kasper Støvring, Henrik Hansen, Troels Krarup PLoS One Research Article Large language models have received enormous attention recently with some studies demonstrating their potential clinical value, despite not being trained specifically for this domain. We aimed to investigate whether ChatGPT, a language model optimized for dialogue, can answer frequently asked questions about diabetes. We conducted a closed e-survey among employees of a large Danish diabetes center. The study design was inspired by the Turing test and non-inferiority trials. Our survey included ten questions with two answers each. One of these was written by a human expert, while the other was generated by ChatGPT. Participants had the task to identify the ChatGPT-generated answer. Data was analyzed at the question-level using logistic regression with robust variance estimation with clustering at participant level. In secondary analyses, we investigated the effect of participant characteristics on the outcome. A 55% non-inferiority margin was pre-defined based on precision simulations and had been published as part of the study protocol before data collection began. Among 311 invited individuals, 183 participated in the survey (59% response rate). 64% had heard of ChatGPT before, and 19% had tried it. Overall, participants could identify ChatGPT-generated answers 59.5% (95% CI: 57.0, 62.0) of the time, which was outside of the non-inferiority zone. Among participant characteristics, previous ChatGPT use had the strongest association with the outcome (odds ratio: 1.52 (1.16, 2.00), p = 0.003). Previous users answered 67.4% (61.7, 72.7) of the questions correctly, versus non-users’ 57.6% (54.9, 60.3). Participants could distinguish between ChatGPT-generated and human-written answers somewhat better than flipping a fair coin, which was against our initial hypothesis. Rigorously planned studies are needed to elucidate the risks and benefits of integrating such technologies in routine clinical practice. Public Library of Science 2023-08-31 /pmc/articles/PMC10470899/ /pubmed/37651381 http://dx.doi.org/10.1371/journal.pone.0290773 Text en © 2023 Hulman et al https://creativecommons.org/licenses/by/4.0/This is an open access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/) , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
spellingShingle Research Article
Hulman, Adam
Dollerup, Ole Lindgård
Mortensen, Jesper Friis
Fenech, Matthew E.
Norman, Kasper
Støvring, Henrik
Hansen, Troels Krarup
ChatGPT- versus human-generated answers to frequently asked questions about diabetes: A Turing test-inspired survey among employees of a Danish diabetes center
title ChatGPT- versus human-generated answers to frequently asked questions about diabetes: A Turing test-inspired survey among employees of a Danish diabetes center
title_full ChatGPT- versus human-generated answers to frequently asked questions about diabetes: A Turing test-inspired survey among employees of a Danish diabetes center
title_fullStr ChatGPT- versus human-generated answers to frequently asked questions about diabetes: A Turing test-inspired survey among employees of a Danish diabetes center
title_full_unstemmed ChatGPT- versus human-generated answers to frequently asked questions about diabetes: A Turing test-inspired survey among employees of a Danish diabetes center
title_short ChatGPT- versus human-generated answers to frequently asked questions about diabetes: A Turing test-inspired survey among employees of a Danish diabetes center
title_sort chatgpt- versus human-generated answers to frequently asked questions about diabetes: a turing test-inspired survey among employees of a danish diabetes center
topic Research Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10470899/
https://www.ncbi.nlm.nih.gov/pubmed/37651381
http://dx.doi.org/10.1371/journal.pone.0290773
work_keys_str_mv AT hulmanadam chatgptversushumangeneratedanswerstofrequentlyaskedquestionsaboutdiabetesaturingtestinspiredsurveyamongemployeesofadanishdiabetescenter
AT dollerupolelindgard chatgptversushumangeneratedanswerstofrequentlyaskedquestionsaboutdiabetesaturingtestinspiredsurveyamongemployeesofadanishdiabetescenter
AT mortensenjesperfriis chatgptversushumangeneratedanswerstofrequentlyaskedquestionsaboutdiabetesaturingtestinspiredsurveyamongemployeesofadanishdiabetescenter
AT fenechmatthewe chatgptversushumangeneratedanswerstofrequentlyaskedquestionsaboutdiabetesaturingtestinspiredsurveyamongemployeesofadanishdiabetescenter
AT normankasper chatgptversushumangeneratedanswerstofrequentlyaskedquestionsaboutdiabetesaturingtestinspiredsurveyamongemployeesofadanishdiabetescenter
AT støvringhenrik chatgptversushumangeneratedanswerstofrequentlyaskedquestionsaboutdiabetesaturingtestinspiredsurveyamongemployeesofadanishdiabetescenter
AT hansentroelskrarup chatgptversushumangeneratedanswerstofrequentlyaskedquestionsaboutdiabetesaturingtestinspiredsurveyamongemployeesofadanishdiabetescenter