Cargando…
Will ChatGPT pass the Polish specialty exam in radiology and diagnostic imaging? Insights into strengths and limitations
PURPOSE: Rapid development of artificial intelligence has aroused curiosity regarding its potential applications in medical field. The purpose of this article was to present the performance of ChatGPT, a state-of-the-art language model in relation to pass rate of national specialty examination (PES)...
Autores principales: | , , , , , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Termedia Publishing House
2023
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10551734/ https://www.ncbi.nlm.nih.gov/pubmed/37808173 http://dx.doi.org/10.5114/pjr.2023.131215 |
_version_ | 1785115835107377152 |
---|---|
author | Kufel, Jakub Paszkiewicz, Iga Bielówka, Michał Bartnikowska, Wiktoria Janik, Michał Stencel, Magdalena Czogalik, Łukasz Gruszczyńska, Katarzyna Mielcarska, Sylwia |
author_facet | Kufel, Jakub Paszkiewicz, Iga Bielówka, Michał Bartnikowska, Wiktoria Janik, Michał Stencel, Magdalena Czogalik, Łukasz Gruszczyńska, Katarzyna Mielcarska, Sylwia |
author_sort | Kufel, Jakub |
collection | PubMed |
description | PURPOSE: Rapid development of artificial intelligence has aroused curiosity regarding its potential applications in medical field. The purpose of this article was to present the performance of ChatGPT, a state-of-the-art language model in relation to pass rate of national specialty examination (PES) in radiology and imaging diagnostics within Polish education system. Additionally, the study aimed to identify the strengths and limitations of the model through a detailed analysis of issues raised by exam questions. MATERIAL AND METHODS: The present study utilized a PES exam consisting of 120 questions, provided by Medical Exami-nations Center in Lodz. Questions were administered using openai.com platform that grants free access to GPT-3.5 model. All questions were categorized according to Bloom’s taxonomy to assess their complexity and difficulty. Following the answer to each exam question, ChatGPT was asked to rate its confidence on a scale of 1 to 5 to evaluate the accuracy of its response. RESULTS: ChatGPT did not reach the pass rate threshold of PES exam (52%); however, it was close in certain question categories. No significant differences were observed in the percentage of correct answers across question types and sub-types. CONCLUSIONS: The performance of the ChatGPT model in the pass rate of PES exam in radiology and imaging diagnostics in Poland is yet to be determined, which requires further research on improved versions of ChatGPT. |
format | Online Article Text |
id | pubmed-10551734 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2023 |
publisher | Termedia Publishing House |
record_format | MEDLINE/PubMed |
spelling | pubmed-105517342023-10-06 Will ChatGPT pass the Polish specialty exam in radiology and diagnostic imaging? Insights into strengths and limitations Kufel, Jakub Paszkiewicz, Iga Bielówka, Michał Bartnikowska, Wiktoria Janik, Michał Stencel, Magdalena Czogalik, Łukasz Gruszczyńska, Katarzyna Mielcarska, Sylwia Pol J Radiol Original Paper PURPOSE: Rapid development of artificial intelligence has aroused curiosity regarding its potential applications in medical field. The purpose of this article was to present the performance of ChatGPT, a state-of-the-art language model in relation to pass rate of national specialty examination (PES) in radiology and imaging diagnostics within Polish education system. Additionally, the study aimed to identify the strengths and limitations of the model through a detailed analysis of issues raised by exam questions. MATERIAL AND METHODS: The present study utilized a PES exam consisting of 120 questions, provided by Medical Exami-nations Center in Lodz. Questions were administered using openai.com platform that grants free access to GPT-3.5 model. All questions were categorized according to Bloom’s taxonomy to assess their complexity and difficulty. Following the answer to each exam question, ChatGPT was asked to rate its confidence on a scale of 1 to 5 to evaluate the accuracy of its response. RESULTS: ChatGPT did not reach the pass rate threshold of PES exam (52%); however, it was close in certain question categories. No significant differences were observed in the percentage of correct answers across question types and sub-types. CONCLUSIONS: The performance of the ChatGPT model in the pass rate of PES exam in radiology and imaging diagnostics in Poland is yet to be determined, which requires further research on improved versions of ChatGPT. Termedia Publishing House 2023-09-18 /pmc/articles/PMC10551734/ /pubmed/37808173 http://dx.doi.org/10.5114/pjr.2023.131215 Text en © Pol J Radiol 2023 https://creativecommons.org/licenses/by-nc-nd/4.0/This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0). License (https://creativecommons.org/licenses/by-nc-nd/4.0/) |
spellingShingle | Original Paper Kufel, Jakub Paszkiewicz, Iga Bielówka, Michał Bartnikowska, Wiktoria Janik, Michał Stencel, Magdalena Czogalik, Łukasz Gruszczyńska, Katarzyna Mielcarska, Sylwia Will ChatGPT pass the Polish specialty exam in radiology and diagnostic imaging? Insights into strengths and limitations |
title | Will ChatGPT pass the Polish specialty exam in radiology and diagnostic imaging? Insights into strengths and limitations |
title_full | Will ChatGPT pass the Polish specialty exam in radiology and diagnostic imaging? Insights into strengths and limitations |
title_fullStr | Will ChatGPT pass the Polish specialty exam in radiology and diagnostic imaging? Insights into strengths and limitations |
title_full_unstemmed | Will ChatGPT pass the Polish specialty exam in radiology and diagnostic imaging? Insights into strengths and limitations |
title_short | Will ChatGPT pass the Polish specialty exam in radiology and diagnostic imaging? Insights into strengths and limitations |
title_sort | will chatgpt pass the polish specialty exam in radiology and diagnostic imaging? insights into strengths and limitations |
topic | Original Paper |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10551734/ https://www.ncbi.nlm.nih.gov/pubmed/37808173 http://dx.doi.org/10.5114/pjr.2023.131215 |
work_keys_str_mv | AT kufeljakub willchatgptpassthepolishspecialtyexaminradiologyanddiagnosticimaginginsightsintostrengthsandlimitations AT paszkiewicziga willchatgptpassthepolishspecialtyexaminradiologyanddiagnosticimaginginsightsintostrengthsandlimitations AT bielowkamichał willchatgptpassthepolishspecialtyexaminradiologyanddiagnosticimaginginsightsintostrengthsandlimitations AT bartnikowskawiktoria willchatgptpassthepolishspecialtyexaminradiologyanddiagnosticimaginginsightsintostrengthsandlimitations AT janikmichał willchatgptpassthepolishspecialtyexaminradiologyanddiagnosticimaginginsightsintostrengthsandlimitations AT stencelmagdalena willchatgptpassthepolishspecialtyexaminradiologyanddiagnosticimaginginsightsintostrengthsandlimitations AT czogalikłukasz willchatgptpassthepolishspecialtyexaminradiologyanddiagnosticimaginginsightsintostrengthsandlimitations AT gruszczynskakatarzyna willchatgptpassthepolishspecialtyexaminradiologyanddiagnosticimaginginsightsintostrengthsandlimitations AT mielcarskasylwia willchatgptpassthepolishspecialtyexaminradiologyanddiagnosticimaginginsightsintostrengthsandlimitations |