Cargando…

Evaluating the Efficacy of ChatGPT in Navigating the Spanish Medical Residency Entrance Examination (MIR): Promising Horizons for AI in Clinical Medicine

The rapid progress in artificial intelligence, machine learning, and natural language processing has led to increasingly sophisticated large language models (LLMs) for use in healthcare. This study assesses the performance of two LLMs, the GPT-3.5 and GPT-4 models, in passing the MIR medical examina...

Descripción completa

Detalles Bibliográficos
Autores principales: Guillen-Grima, Francisco, Guillen-Aguinaga, Sara, Guillen-Aguinaga, Laura, Alas-Brun, Rosa, Onambele, Luc, Ortega, Wilfrido, Montejo, Rocio, Aguinaga-Ontoso, Enrique, Barach, Paul, Aguinaga-Ontoso, Ines
Formato: Online Artículo Texto
Lenguaje:English
Publicado: MDPI 2023
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10660543/
https://www.ncbi.nlm.nih.gov/pubmed/37987431
http://dx.doi.org/10.3390/clinpract13060130
_version_ 1785148413092823040
author Guillen-Grima, Francisco
Guillen-Aguinaga, Sara
Guillen-Aguinaga, Laura
Alas-Brun, Rosa
Onambele, Luc
Ortega, Wilfrido
Montejo, Rocio
Aguinaga-Ontoso, Enrique
Barach, Paul
Aguinaga-Ontoso, Ines
author_facet Guillen-Grima, Francisco
Guillen-Aguinaga, Sara
Guillen-Aguinaga, Laura
Alas-Brun, Rosa
Onambele, Luc
Ortega, Wilfrido
Montejo, Rocio
Aguinaga-Ontoso, Enrique
Barach, Paul
Aguinaga-Ontoso, Ines
author_sort Guillen-Grima, Francisco
collection PubMed
description The rapid progress in artificial intelligence, machine learning, and natural language processing has led to increasingly sophisticated large language models (LLMs) for use in healthcare. This study assesses the performance of two LLMs, the GPT-3.5 and GPT-4 models, in passing the MIR medical examination for access to medical specialist training in Spain. Our objectives included gauging the model’s overall performance, analyzing discrepancies across different medical specialties, discerning between theoretical and practical questions, estimating error proportions, and assessing the hypothetical severity of errors committed by a physician. Material and methods: We studied the 2022 Spanish MIR examination results after excluding those questions requiring image evaluations or having acknowledged errors. The remaining 182 questions were presented to the LLM GPT-4 and GPT-3.5 in Spanish and English. Logistic regression models analyzed the relationships between question length, sequence, and performance. We also analyzed the 23 questions with images, using GPT-4’s new image analysis capability. Results: GPT-4 outperformed GPT-3.5, scoring 86.81% in Spanish (p < 0.001). English translations had a slightly enhanced performance. GPT-4 scored 26.1% of the questions with images in English. The results were worse when the questions were in Spanish, 13.0%, although the differences were not statistically significant (p = 0.250). Among medical specialties, GPT-4 achieved a 100% correct response rate in several areas, and the Pharmacology, Critical Care, and Infectious Diseases specialties showed lower performance. The error analysis revealed that while a 13.2% error rate existed, the gravest categories, such as “error requiring intervention to sustain life” and “error resulting in death”, had a 0% rate. Conclusions: GPT-4 performs robustly on the Spanish MIR examination, with varying capabilities to discriminate knowledge across specialties. While the model’s high success rate is commendable, understanding the error severity is critical, especially when considering AI’s potential role in real-world medical practice and its implications for patient safety.
format Online
Article
Text
id pubmed-10660543
institution National Center for Biotechnology Information
language English
publishDate 2023
publisher MDPI
record_format MEDLINE/PubMed
spelling pubmed-106605432023-11-20 Evaluating the Efficacy of ChatGPT in Navigating the Spanish Medical Residency Entrance Examination (MIR): Promising Horizons for AI in Clinical Medicine Guillen-Grima, Francisco Guillen-Aguinaga, Sara Guillen-Aguinaga, Laura Alas-Brun, Rosa Onambele, Luc Ortega, Wilfrido Montejo, Rocio Aguinaga-Ontoso, Enrique Barach, Paul Aguinaga-Ontoso, Ines Clin Pract Article The rapid progress in artificial intelligence, machine learning, and natural language processing has led to increasingly sophisticated large language models (LLMs) for use in healthcare. This study assesses the performance of two LLMs, the GPT-3.5 and GPT-4 models, in passing the MIR medical examination for access to medical specialist training in Spain. Our objectives included gauging the model’s overall performance, analyzing discrepancies across different medical specialties, discerning between theoretical and practical questions, estimating error proportions, and assessing the hypothetical severity of errors committed by a physician. Material and methods: We studied the 2022 Spanish MIR examination results after excluding those questions requiring image evaluations or having acknowledged errors. The remaining 182 questions were presented to the LLM GPT-4 and GPT-3.5 in Spanish and English. Logistic regression models analyzed the relationships between question length, sequence, and performance. We also analyzed the 23 questions with images, using GPT-4’s new image analysis capability. Results: GPT-4 outperformed GPT-3.5, scoring 86.81% in Spanish (p < 0.001). English translations had a slightly enhanced performance. GPT-4 scored 26.1% of the questions with images in English. The results were worse when the questions were in Spanish, 13.0%, although the differences were not statistically significant (p = 0.250). Among medical specialties, GPT-4 achieved a 100% correct response rate in several areas, and the Pharmacology, Critical Care, and Infectious Diseases specialties showed lower performance. The error analysis revealed that while a 13.2% error rate existed, the gravest categories, such as “error requiring intervention to sustain life” and “error resulting in death”, had a 0% rate. Conclusions: GPT-4 performs robustly on the Spanish MIR examination, with varying capabilities to discriminate knowledge across specialties. While the model’s high success rate is commendable, understanding the error severity is critical, especially when considering AI’s potential role in real-world medical practice and its implications for patient safety. MDPI 2023-11-20 /pmc/articles/PMC10660543/ /pubmed/37987431 http://dx.doi.org/10.3390/clinpract13060130 Text en © 2023 by the authors. https://creativecommons.org/licenses/by/4.0/Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
spellingShingle Article
Guillen-Grima, Francisco
Guillen-Aguinaga, Sara
Guillen-Aguinaga, Laura
Alas-Brun, Rosa
Onambele, Luc
Ortega, Wilfrido
Montejo, Rocio
Aguinaga-Ontoso, Enrique
Barach, Paul
Aguinaga-Ontoso, Ines
Evaluating the Efficacy of ChatGPT in Navigating the Spanish Medical Residency Entrance Examination (MIR): Promising Horizons for AI in Clinical Medicine
title Evaluating the Efficacy of ChatGPT in Navigating the Spanish Medical Residency Entrance Examination (MIR): Promising Horizons for AI in Clinical Medicine
title_full Evaluating the Efficacy of ChatGPT in Navigating the Spanish Medical Residency Entrance Examination (MIR): Promising Horizons for AI in Clinical Medicine
title_fullStr Evaluating the Efficacy of ChatGPT in Navigating the Spanish Medical Residency Entrance Examination (MIR): Promising Horizons for AI in Clinical Medicine
title_full_unstemmed Evaluating the Efficacy of ChatGPT in Navigating the Spanish Medical Residency Entrance Examination (MIR): Promising Horizons for AI in Clinical Medicine
title_short Evaluating the Efficacy of ChatGPT in Navigating the Spanish Medical Residency Entrance Examination (MIR): Promising Horizons for AI in Clinical Medicine
title_sort evaluating the efficacy of chatgpt in navigating the spanish medical residency entrance examination (mir): promising horizons for ai in clinical medicine
topic Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10660543/
https://www.ncbi.nlm.nih.gov/pubmed/37987431
http://dx.doi.org/10.3390/clinpract13060130
work_keys_str_mv AT guillengrimafrancisco evaluatingtheefficacyofchatgptinnavigatingthespanishmedicalresidencyentranceexaminationmirpromisinghorizonsforaiinclinicalmedicine
AT guillenaguinagasara evaluatingtheefficacyofchatgptinnavigatingthespanishmedicalresidencyentranceexaminationmirpromisinghorizonsforaiinclinicalmedicine
AT guillenaguinagalaura evaluatingtheefficacyofchatgptinnavigatingthespanishmedicalresidencyentranceexaminationmirpromisinghorizonsforaiinclinicalmedicine
AT alasbrunrosa evaluatingtheefficacyofchatgptinnavigatingthespanishmedicalresidencyentranceexaminationmirpromisinghorizonsforaiinclinicalmedicine
AT onambeleluc evaluatingtheefficacyofchatgptinnavigatingthespanishmedicalresidencyentranceexaminationmirpromisinghorizonsforaiinclinicalmedicine
AT ortegawilfrido evaluatingtheefficacyofchatgptinnavigatingthespanishmedicalresidencyentranceexaminationmirpromisinghorizonsforaiinclinicalmedicine
AT montejorocio evaluatingtheefficacyofchatgptinnavigatingthespanishmedicalresidencyentranceexaminationmirpromisinghorizonsforaiinclinicalmedicine
AT aguinagaontosoenrique evaluatingtheefficacyofchatgptinnavigatingthespanishmedicalresidencyentranceexaminationmirpromisinghorizonsforaiinclinicalmedicine
AT barachpaul evaluatingtheefficacyofchatgptinnavigatingthespanishmedicalresidencyentranceexaminationmirpromisinghorizonsforaiinclinicalmedicine
AT aguinagaontosoines evaluatingtheefficacyofchatgptinnavigatingthespanishmedicalresidencyentranceexaminationmirpromisinghorizonsforaiinclinicalmedicine