Cargando…
Design Characteristics of Studies Reporting the Performance of Artificial Intelligence Algorithms for Diagnostic Analysis of Medical Images: Results from Recently Published Papers
OBJECTIVE: To evaluate the design characteristics of studies that evaluated the performance of artificial intelligence (AI) algorithms for the diagnostic analysis of medical images. MATERIALS AND METHODS: PubMed MEDLINE and Embase databases were searched to identify original research articles publis...
Autores principales: | , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
The Korean Society of Radiology
2019
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6389801/ https://www.ncbi.nlm.nih.gov/pubmed/30799571 http://dx.doi.org/10.3348/kjr.2019.0025 |
_version_ | 1783398005296070656 |
---|---|
author | Kim, Dong Wook Jang, Hye Young Kim, Kyung Won Shin, Youngbin Park, Seong Ho |
author_facet | Kim, Dong Wook Jang, Hye Young Kim, Kyung Won Shin, Youngbin Park, Seong Ho |
author_sort | Kim, Dong Wook |
collection | PubMed |
description | OBJECTIVE: To evaluate the design characteristics of studies that evaluated the performance of artificial intelligence (AI) algorithms for the diagnostic analysis of medical images. MATERIALS AND METHODS: PubMed MEDLINE and Embase databases were searched to identify original research articles published between January 1, 2018 and August 17, 2018 that investigated the performance of AI algorithms that analyze medical images to provide diagnostic decisions. Eligible articles were evaluated to determine 1) whether the study used external validation rather than internal validation, and in case of external validation, whether the data for validation were collected, 2) with diagnostic cohort design instead of diagnostic case-control design, 3) from multiple institutions, and 4) in a prospective manner. These are fundamental methodologic features recommended for clinical validation of AI performance in real-world practice. The studies that fulfilled the above criteria were identified. We classified the publishing journals into medical vs. non-medical journal groups. Then, the results were compared between medical and non-medical journals. RESULTS: Of 516 eligible published studies, only 6% (31 studies) performed external validation. None of the 31 studies adopted all three design features: diagnostic cohort design, the inclusion of multiple institutions, and prospective data collection for external validation. No significant difference was found between medical and non-medical journals. CONCLUSION: Nearly all of the studies published in the study period that evaluated the performance of AI algorithms for diagnostic analysis of medical images were designed as proof-of-concept technical feasibility studies and did not have the design features that are recommended for robust validation of the real-world clinical performance of AI algorithms. |
format | Online Article Text |
id | pubmed-6389801 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2019 |
publisher | The Korean Society of Radiology |
record_format | MEDLINE/PubMed |
spelling | pubmed-63898012019-03-05 Design Characteristics of Studies Reporting the Performance of Artificial Intelligence Algorithms for Diagnostic Analysis of Medical Images: Results from Recently Published Papers Kim, Dong Wook Jang, Hye Young Kim, Kyung Won Shin, Youngbin Park, Seong Ho Korean J Radiol Artificial Intelligence OBJECTIVE: To evaluate the design characteristics of studies that evaluated the performance of artificial intelligence (AI) algorithms for the diagnostic analysis of medical images. MATERIALS AND METHODS: PubMed MEDLINE and Embase databases were searched to identify original research articles published between January 1, 2018 and August 17, 2018 that investigated the performance of AI algorithms that analyze medical images to provide diagnostic decisions. Eligible articles were evaluated to determine 1) whether the study used external validation rather than internal validation, and in case of external validation, whether the data for validation were collected, 2) with diagnostic cohort design instead of diagnostic case-control design, 3) from multiple institutions, and 4) in a prospective manner. These are fundamental methodologic features recommended for clinical validation of AI performance in real-world practice. The studies that fulfilled the above criteria were identified. We classified the publishing journals into medical vs. non-medical journal groups. Then, the results were compared between medical and non-medical journals. RESULTS: Of 516 eligible published studies, only 6% (31 studies) performed external validation. None of the 31 studies adopted all three design features: diagnostic cohort design, the inclusion of multiple institutions, and prospective data collection for external validation. No significant difference was found between medical and non-medical journals. CONCLUSION: Nearly all of the studies published in the study period that evaluated the performance of AI algorithms for diagnostic analysis of medical images were designed as proof-of-concept technical feasibility studies and did not have the design features that are recommended for robust validation of the real-world clinical performance of AI algorithms. The Korean Society of Radiology 2019-03 2019-02-19 /pmc/articles/PMC6389801/ /pubmed/30799571 http://dx.doi.org/10.3348/kjr.2019.0025 Text en Copyright © 2019 The Korean Society of Radiology http://creativecommons.org/licenses/by-nc/4.0/ This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0/) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited. |
spellingShingle | Artificial Intelligence Kim, Dong Wook Jang, Hye Young Kim, Kyung Won Shin, Youngbin Park, Seong Ho Design Characteristics of Studies Reporting the Performance of Artificial Intelligence Algorithms for Diagnostic Analysis of Medical Images: Results from Recently Published Papers |
title | Design Characteristics of Studies Reporting the Performance of Artificial Intelligence Algorithms for Diagnostic Analysis of Medical Images: Results from Recently Published Papers |
title_full | Design Characteristics of Studies Reporting the Performance of Artificial Intelligence Algorithms for Diagnostic Analysis of Medical Images: Results from Recently Published Papers |
title_fullStr | Design Characteristics of Studies Reporting the Performance of Artificial Intelligence Algorithms for Diagnostic Analysis of Medical Images: Results from Recently Published Papers |
title_full_unstemmed | Design Characteristics of Studies Reporting the Performance of Artificial Intelligence Algorithms for Diagnostic Analysis of Medical Images: Results from Recently Published Papers |
title_short | Design Characteristics of Studies Reporting the Performance of Artificial Intelligence Algorithms for Diagnostic Analysis of Medical Images: Results from Recently Published Papers |
title_sort | design characteristics of studies reporting the performance of artificial intelligence algorithms for diagnostic analysis of medical images: results from recently published papers |
topic | Artificial Intelligence |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6389801/ https://www.ncbi.nlm.nih.gov/pubmed/30799571 http://dx.doi.org/10.3348/kjr.2019.0025 |
work_keys_str_mv | AT kimdongwook designcharacteristicsofstudiesreportingtheperformanceofartificialintelligencealgorithmsfordiagnosticanalysisofmedicalimagesresultsfromrecentlypublishedpapers AT janghyeyoung designcharacteristicsofstudiesreportingtheperformanceofartificialintelligencealgorithmsfordiagnosticanalysisofmedicalimagesresultsfromrecentlypublishedpapers AT kimkyungwon designcharacteristicsofstudiesreportingtheperformanceofartificialintelligencealgorithmsfordiagnosticanalysisofmedicalimagesresultsfromrecentlypublishedpapers AT shinyoungbin designcharacteristicsofstudiesreportingtheperformanceofartificialintelligencealgorithmsfordiagnosticanalysisofmedicalimagesresultsfromrecentlypublishedpapers AT parkseongho designcharacteristicsofstudiesreportingtheperformanceofartificialintelligencealgorithmsfordiagnosticanalysisofmedicalimagesresultsfromrecentlypublishedpapers |