Cargando…
ChatGPT- versus human-generated answers to frequently asked questions about diabetes: A Turing test-inspired survey among employees of a Danish diabetes center
Large language models have received enormous attention recently with some studies demonstrating their potential clinical value, despite not being trained specifically for this domain. We aimed to investigate whether ChatGPT, a language model optimized for dialogue, can answer frequently asked questi...
Autores principales: | , , , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Public Library of Science
2023
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10470899/ https://www.ncbi.nlm.nih.gov/pubmed/37651381 http://dx.doi.org/10.1371/journal.pone.0290773 |
_version_ | 1785099784710782976 |
---|---|
author | Hulman, Adam Dollerup, Ole Lindgård Mortensen, Jesper Friis Fenech, Matthew E. Norman, Kasper Støvring, Henrik Hansen, Troels Krarup |
author_facet | Hulman, Adam Dollerup, Ole Lindgård Mortensen, Jesper Friis Fenech, Matthew E. Norman, Kasper Støvring, Henrik Hansen, Troels Krarup |
author_sort | Hulman, Adam |
collection | PubMed |
description | Large language models have received enormous attention recently with some studies demonstrating their potential clinical value, despite not being trained specifically for this domain. We aimed to investigate whether ChatGPT, a language model optimized for dialogue, can answer frequently asked questions about diabetes. We conducted a closed e-survey among employees of a large Danish diabetes center. The study design was inspired by the Turing test and non-inferiority trials. Our survey included ten questions with two answers each. One of these was written by a human expert, while the other was generated by ChatGPT. Participants had the task to identify the ChatGPT-generated answer. Data was analyzed at the question-level using logistic regression with robust variance estimation with clustering at participant level. In secondary analyses, we investigated the effect of participant characteristics on the outcome. A 55% non-inferiority margin was pre-defined based on precision simulations and had been published as part of the study protocol before data collection began. Among 311 invited individuals, 183 participated in the survey (59% response rate). 64% had heard of ChatGPT before, and 19% had tried it. Overall, participants could identify ChatGPT-generated answers 59.5% (95% CI: 57.0, 62.0) of the time, which was outside of the non-inferiority zone. Among participant characteristics, previous ChatGPT use had the strongest association with the outcome (odds ratio: 1.52 (1.16, 2.00), p = 0.003). Previous users answered 67.4% (61.7, 72.7) of the questions correctly, versus non-users’ 57.6% (54.9, 60.3). Participants could distinguish between ChatGPT-generated and human-written answers somewhat better than flipping a fair coin, which was against our initial hypothesis. Rigorously planned studies are needed to elucidate the risks and benefits of integrating such technologies in routine clinical practice. |
format | Online Article Text |
id | pubmed-10470899 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2023 |
publisher | Public Library of Science |
record_format | MEDLINE/PubMed |
spelling | pubmed-104708992023-09-01 ChatGPT- versus human-generated answers to frequently asked questions about diabetes: A Turing test-inspired survey among employees of a Danish diabetes center Hulman, Adam Dollerup, Ole Lindgård Mortensen, Jesper Friis Fenech, Matthew E. Norman, Kasper Støvring, Henrik Hansen, Troels Krarup PLoS One Research Article Large language models have received enormous attention recently with some studies demonstrating their potential clinical value, despite not being trained specifically for this domain. We aimed to investigate whether ChatGPT, a language model optimized for dialogue, can answer frequently asked questions about diabetes. We conducted a closed e-survey among employees of a large Danish diabetes center. The study design was inspired by the Turing test and non-inferiority trials. Our survey included ten questions with two answers each. One of these was written by a human expert, while the other was generated by ChatGPT. Participants had the task to identify the ChatGPT-generated answer. Data was analyzed at the question-level using logistic regression with robust variance estimation with clustering at participant level. In secondary analyses, we investigated the effect of participant characteristics on the outcome. A 55% non-inferiority margin was pre-defined based on precision simulations and had been published as part of the study protocol before data collection began. Among 311 invited individuals, 183 participated in the survey (59% response rate). 64% had heard of ChatGPT before, and 19% had tried it. Overall, participants could identify ChatGPT-generated answers 59.5% (95% CI: 57.0, 62.0) of the time, which was outside of the non-inferiority zone. Among participant characteristics, previous ChatGPT use had the strongest association with the outcome (odds ratio: 1.52 (1.16, 2.00), p = 0.003). Previous users answered 67.4% (61.7, 72.7) of the questions correctly, versus non-users’ 57.6% (54.9, 60.3). Participants could distinguish between ChatGPT-generated and human-written answers somewhat better than flipping a fair coin, which was against our initial hypothesis. Rigorously planned studies are needed to elucidate the risks and benefits of integrating such technologies in routine clinical practice. Public Library of Science 2023-08-31 /pmc/articles/PMC10470899/ /pubmed/37651381 http://dx.doi.org/10.1371/journal.pone.0290773 Text en © 2023 Hulman et al https://creativecommons.org/licenses/by/4.0/This is an open access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/) , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. |
spellingShingle | Research Article Hulman, Adam Dollerup, Ole Lindgård Mortensen, Jesper Friis Fenech, Matthew E. Norman, Kasper Støvring, Henrik Hansen, Troels Krarup ChatGPT- versus human-generated answers to frequently asked questions about diabetes: A Turing test-inspired survey among employees of a Danish diabetes center |
title | ChatGPT- versus human-generated answers to frequently asked questions about diabetes: A Turing test-inspired survey among employees of a Danish diabetes center |
title_full | ChatGPT- versus human-generated answers to frequently asked questions about diabetes: A Turing test-inspired survey among employees of a Danish diabetes center |
title_fullStr | ChatGPT- versus human-generated answers to frequently asked questions about diabetes: A Turing test-inspired survey among employees of a Danish diabetes center |
title_full_unstemmed | ChatGPT- versus human-generated answers to frequently asked questions about diabetes: A Turing test-inspired survey among employees of a Danish diabetes center |
title_short | ChatGPT- versus human-generated answers to frequently asked questions about diabetes: A Turing test-inspired survey among employees of a Danish diabetes center |
title_sort | chatgpt- versus human-generated answers to frequently asked questions about diabetes: a turing test-inspired survey among employees of a danish diabetes center |
topic | Research Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10470899/ https://www.ncbi.nlm.nih.gov/pubmed/37651381 http://dx.doi.org/10.1371/journal.pone.0290773 |
work_keys_str_mv | AT hulmanadam chatgptversushumangeneratedanswerstofrequentlyaskedquestionsaboutdiabetesaturingtestinspiredsurveyamongemployeesofadanishdiabetescenter AT dollerupolelindgard chatgptversushumangeneratedanswerstofrequentlyaskedquestionsaboutdiabetesaturingtestinspiredsurveyamongemployeesofadanishdiabetescenter AT mortensenjesperfriis chatgptversushumangeneratedanswerstofrequentlyaskedquestionsaboutdiabetesaturingtestinspiredsurveyamongemployeesofadanishdiabetescenter AT fenechmatthewe chatgptversushumangeneratedanswerstofrequentlyaskedquestionsaboutdiabetesaturingtestinspiredsurveyamongemployeesofadanishdiabetescenter AT normankasper chatgptversushumangeneratedanswerstofrequentlyaskedquestionsaboutdiabetesaturingtestinspiredsurveyamongemployeesofadanishdiabetescenter AT støvringhenrik chatgptversushumangeneratedanswerstofrequentlyaskedquestionsaboutdiabetesaturingtestinspiredsurveyamongemployeesofadanishdiabetescenter AT hansentroelskrarup chatgptversushumangeneratedanswerstofrequentlyaskedquestionsaboutdiabetesaturingtestinspiredsurveyamongemployeesofadanishdiabetescenter |