Cargando…

Will ChatGPT pass the Polish specialty exam in radiology and diagnostic imaging? Insights into strengths and limitations

PURPOSE: Rapid development of artificial intelligence has aroused curiosity regarding its potential applications in medical field. The purpose of this article was to present the performance of ChatGPT, a state-of-the-art language model in relation to pass rate of national specialty examination (PES)...

Descripción completa

Detalles Bibliográficos
Autores principales: Kufel, Jakub, Paszkiewicz, Iga, Bielówka, Michał, Bartnikowska, Wiktoria, Janik, Michał, Stencel, Magdalena, Czogalik, Łukasz, Gruszczyńska, Katarzyna, Mielcarska, Sylwia
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Termedia Publishing House 2023
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10551734/
https://www.ncbi.nlm.nih.gov/pubmed/37808173
http://dx.doi.org/10.5114/pjr.2023.131215
_version_ 1785115835107377152
author Kufel, Jakub
Paszkiewicz, Iga
Bielówka, Michał
Bartnikowska, Wiktoria
Janik, Michał
Stencel, Magdalena
Czogalik, Łukasz
Gruszczyńska, Katarzyna
Mielcarska, Sylwia
author_facet Kufel, Jakub
Paszkiewicz, Iga
Bielówka, Michał
Bartnikowska, Wiktoria
Janik, Michał
Stencel, Magdalena
Czogalik, Łukasz
Gruszczyńska, Katarzyna
Mielcarska, Sylwia
author_sort Kufel, Jakub
collection PubMed
description PURPOSE: Rapid development of artificial intelligence has aroused curiosity regarding its potential applications in medical field. The purpose of this article was to present the performance of ChatGPT, a state-of-the-art language model in relation to pass rate of national specialty examination (PES) in radiology and imaging diagnostics within Polish education system. Additionally, the study aimed to identify the strengths and limitations of the model through a detailed analysis of issues raised by exam questions. MATERIAL AND METHODS: The present study utilized a PES exam consisting of 120 questions, provided by Medical Exami-nations Center in Lodz. Questions were administered using openai.com platform that grants free access to GPT-3.5 model. All questions were categorized according to Bloom’s taxonomy to assess their complexity and difficulty. Following the answer to each exam question, ChatGPT was asked to rate its confidence on a scale of 1 to 5 to evaluate the accuracy of its response. RESULTS: ChatGPT did not reach the pass rate threshold of PES exam (52%); however, it was close in certain question categories. No significant differences were observed in the percentage of correct answers across question types and sub-types. CONCLUSIONS: The performance of the ChatGPT model in the pass rate of PES exam in radiology and imaging diagnostics in Poland is yet to be determined, which requires further research on improved versions of ChatGPT.
format Online
Article
Text
id pubmed-10551734
institution National Center for Biotechnology Information
language English
publishDate 2023
publisher Termedia Publishing House
record_format MEDLINE/PubMed
spelling pubmed-105517342023-10-06 Will ChatGPT pass the Polish specialty exam in radiology and diagnostic imaging? Insights into strengths and limitations Kufel, Jakub Paszkiewicz, Iga Bielówka, Michał Bartnikowska, Wiktoria Janik, Michał Stencel, Magdalena Czogalik, Łukasz Gruszczyńska, Katarzyna Mielcarska, Sylwia Pol J Radiol Original Paper PURPOSE: Rapid development of artificial intelligence has aroused curiosity regarding its potential applications in medical field. The purpose of this article was to present the performance of ChatGPT, a state-of-the-art language model in relation to pass rate of national specialty examination (PES) in radiology and imaging diagnostics within Polish education system. Additionally, the study aimed to identify the strengths and limitations of the model through a detailed analysis of issues raised by exam questions. MATERIAL AND METHODS: The present study utilized a PES exam consisting of 120 questions, provided by Medical Exami-nations Center in Lodz. Questions were administered using openai.com platform that grants free access to GPT-3.5 model. All questions were categorized according to Bloom’s taxonomy to assess their complexity and difficulty. Following the answer to each exam question, ChatGPT was asked to rate its confidence on a scale of 1 to 5 to evaluate the accuracy of its response. RESULTS: ChatGPT did not reach the pass rate threshold of PES exam (52%); however, it was close in certain question categories. No significant differences were observed in the percentage of correct answers across question types and sub-types. CONCLUSIONS: The performance of the ChatGPT model in the pass rate of PES exam in radiology and imaging diagnostics in Poland is yet to be determined, which requires further research on improved versions of ChatGPT. Termedia Publishing House 2023-09-18 /pmc/articles/PMC10551734/ /pubmed/37808173 http://dx.doi.org/10.5114/pjr.2023.131215 Text en © Pol J Radiol 2023 https://creativecommons.org/licenses/by-nc-nd/4.0/This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0). License (https://creativecommons.org/licenses/by-nc-nd/4.0/)
spellingShingle Original Paper
Kufel, Jakub
Paszkiewicz, Iga
Bielówka, Michał
Bartnikowska, Wiktoria
Janik, Michał
Stencel, Magdalena
Czogalik, Łukasz
Gruszczyńska, Katarzyna
Mielcarska, Sylwia
Will ChatGPT pass the Polish specialty exam in radiology and diagnostic imaging? Insights into strengths and limitations
title Will ChatGPT pass the Polish specialty exam in radiology and diagnostic imaging? Insights into strengths and limitations
title_full Will ChatGPT pass the Polish specialty exam in radiology and diagnostic imaging? Insights into strengths and limitations
title_fullStr Will ChatGPT pass the Polish specialty exam in radiology and diagnostic imaging? Insights into strengths and limitations
title_full_unstemmed Will ChatGPT pass the Polish specialty exam in radiology and diagnostic imaging? Insights into strengths and limitations
title_short Will ChatGPT pass the Polish specialty exam in radiology and diagnostic imaging? Insights into strengths and limitations
title_sort will chatgpt pass the polish specialty exam in radiology and diagnostic imaging? insights into strengths and limitations
topic Original Paper
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10551734/
https://www.ncbi.nlm.nih.gov/pubmed/37808173
http://dx.doi.org/10.5114/pjr.2023.131215
work_keys_str_mv AT kufeljakub willchatgptpassthepolishspecialtyexaminradiologyanddiagnosticimaginginsightsintostrengthsandlimitations
AT paszkiewicziga willchatgptpassthepolishspecialtyexaminradiologyanddiagnosticimaginginsightsintostrengthsandlimitations
AT bielowkamichał willchatgptpassthepolishspecialtyexaminradiologyanddiagnosticimaginginsightsintostrengthsandlimitations
AT bartnikowskawiktoria willchatgptpassthepolishspecialtyexaminradiologyanddiagnosticimaginginsightsintostrengthsandlimitations
AT janikmichał willchatgptpassthepolishspecialtyexaminradiologyanddiagnosticimaginginsightsintostrengthsandlimitations
AT stencelmagdalena willchatgptpassthepolishspecialtyexaminradiologyanddiagnosticimaginginsightsintostrengthsandlimitations
AT czogalikłukasz willchatgptpassthepolishspecialtyexaminradiologyanddiagnosticimaginginsightsintostrengthsandlimitations
AT gruszczynskakatarzyna willchatgptpassthepolishspecialtyexaminradiologyanddiagnosticimaginginsightsintostrengthsandlimitations
AT mielcarskasylwia willchatgptpassthepolishspecialtyexaminradiologyanddiagnosticimaginginsightsintostrengthsandlimitations