Cargando…

External COVID-19 Deep Learning Model Validation on ACR AI-LAB: It’s a Brave New World

PURPOSE: Deploying external artificial intelligence (AI) models locally can be logistically challenging. We aimed to use the ACR AI-LAB software platform for local testing of a chest radiograph (CXR) algorithm for COVID-19 lung disease severity assessment. METHODS: An externally developed deep learn...

Descripción completa

Detalles Bibliográficos
Autores principales: Ardestani, Ali, Li, Matthew D., Chea, Pauley, Wortman, Jeremy R., Medina, Adam, Kalpathy-Cramer, Jayashree, Wald, Christoph
Formato: Online Artículo Texto
Lenguaje:English
Publicado: American College of Radiology 2022
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8989698/
https://www.ncbi.nlm.nih.gov/pubmed/35483438
http://dx.doi.org/10.1016/j.jacr.2022.03.013
_version_ 1784683228745957376
author Ardestani, Ali
Li, Matthew D.
Chea, Pauley
Wortman, Jeremy R.
Medina, Adam
Kalpathy-Cramer, Jayashree
Wald, Christoph
author_facet Ardestani, Ali
Li, Matthew D.
Chea, Pauley
Wortman, Jeremy R.
Medina, Adam
Kalpathy-Cramer, Jayashree
Wald, Christoph
author_sort Ardestani, Ali
collection PubMed
description PURPOSE: Deploying external artificial intelligence (AI) models locally can be logistically challenging. We aimed to use the ACR AI-LAB software platform for local testing of a chest radiograph (CXR) algorithm for COVID-19 lung disease severity assessment. METHODS: An externally developed deep learning model for COVID-19 radiographic lung disease severity assessment was loaded into the AI-LAB platform at an independent academic medical center, which was separate from the institution in which the model was trained. The data set consisted of CXR images from 141 patients with reverse transcription-polymerase chain reaction–confirmed COVID-19, which were routed to AI-LAB for model inference. The model calculated a Pulmonary X-ray Severity (PXS) score for each image. This score was correlated with the average of a radiologist-based assessment of severity, the modified Radiographic Assessment of Lung Edema score, independently interpreted by three radiologists. The associations between the PXS score and patient admission and intubation or death were assessed. RESULTS: The PXS score deployed in AI-LAB correlated with the radiologist-determined modified Radiographic Assessment of Lung Edema score (r = 0.80). PXS score was significantly higher in patients who were admitted (4.0 versus 1.3, P < .001) or intubated or died within 3 days (5.5 versus 3.3, P = .001). CONCLUSIONS: AI-LAB was successfully used to test an external COVID-19 CXR AI algorithm on local data with relative ease, showing generalizability of the PXS score model. For AI models to scale and be clinically useful, software tools that facilitate the local testing process, like the freely available AI-LAB, will be important to cross the AI implementation gap in health care systems.
format Online
Article
Text
id pubmed-8989698
institution National Center for Biotechnology Information
language English
publishDate 2022
publisher American College of Radiology
record_format MEDLINE/PubMed
spelling pubmed-89896982022-04-11 External COVID-19 Deep Learning Model Validation on ACR AI-LAB: It’s a Brave New World Ardestani, Ali Li, Matthew D. Chea, Pauley Wortman, Jeremy R. Medina, Adam Kalpathy-Cramer, Jayashree Wald, Christoph J Am Coll Radiol Data Science PURPOSE: Deploying external artificial intelligence (AI) models locally can be logistically challenging. We aimed to use the ACR AI-LAB software platform for local testing of a chest radiograph (CXR) algorithm for COVID-19 lung disease severity assessment. METHODS: An externally developed deep learning model for COVID-19 radiographic lung disease severity assessment was loaded into the AI-LAB platform at an independent academic medical center, which was separate from the institution in which the model was trained. The data set consisted of CXR images from 141 patients with reverse transcription-polymerase chain reaction–confirmed COVID-19, which were routed to AI-LAB for model inference. The model calculated a Pulmonary X-ray Severity (PXS) score for each image. This score was correlated with the average of a radiologist-based assessment of severity, the modified Radiographic Assessment of Lung Edema score, independently interpreted by three radiologists. The associations between the PXS score and patient admission and intubation or death were assessed. RESULTS: The PXS score deployed in AI-LAB correlated with the radiologist-determined modified Radiographic Assessment of Lung Edema score (r = 0.80). PXS score was significantly higher in patients who were admitted (4.0 versus 1.3, P < .001) or intubated or died within 3 days (5.5 versus 3.3, P = .001). CONCLUSIONS: AI-LAB was successfully used to test an external COVID-19 CXR AI algorithm on local data with relative ease, showing generalizability of the PXS score model. For AI models to scale and be clinically useful, software tools that facilitate the local testing process, like the freely available AI-LAB, will be important to cross the AI implementation gap in health care systems. American College of Radiology 2022-07 2022-04-08 /pmc/articles/PMC8989698/ /pubmed/35483438 http://dx.doi.org/10.1016/j.jacr.2022.03.013 Text en © 2022 American College of Radiology. Since January 2020 Elsevier has created a COVID-19 resource centre with free information in English and Mandarin on the novel coronavirus COVID-19. The COVID-19 resource centre is hosted on Elsevier Connect, the company's public news and information website. Elsevier hereby grants permission to make all its COVID-19-related research that is available on the COVID-19 resource centre - including this research content - immediately available in PubMed Central and other publicly funded repositories, such as the WHO COVID database with rights for unrestricted research re-use and analyses in any form or by any means with acknowledgement of the original source. These permissions are granted for free by Elsevier for as long as the COVID-19 resource centre remains active.
spellingShingle Data Science
Ardestani, Ali
Li, Matthew D.
Chea, Pauley
Wortman, Jeremy R.
Medina, Adam
Kalpathy-Cramer, Jayashree
Wald, Christoph
External COVID-19 Deep Learning Model Validation on ACR AI-LAB: It’s a Brave New World
title External COVID-19 Deep Learning Model Validation on ACR AI-LAB: It’s a Brave New World
title_full External COVID-19 Deep Learning Model Validation on ACR AI-LAB: It’s a Brave New World
title_fullStr External COVID-19 Deep Learning Model Validation on ACR AI-LAB: It’s a Brave New World
title_full_unstemmed External COVID-19 Deep Learning Model Validation on ACR AI-LAB: It’s a Brave New World
title_short External COVID-19 Deep Learning Model Validation on ACR AI-LAB: It’s a Brave New World
title_sort external covid-19 deep learning model validation on acr ai-lab: it’s a brave new world
topic Data Science
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8989698/
https://www.ncbi.nlm.nih.gov/pubmed/35483438
http://dx.doi.org/10.1016/j.jacr.2022.03.013
work_keys_str_mv AT ardestaniali externalcovid19deeplearningmodelvalidationonacrailabitsabravenewworld
AT limatthewd externalcovid19deeplearningmodelvalidationonacrailabitsabravenewworld
AT cheapauley externalcovid19deeplearningmodelvalidationonacrailabitsabravenewworld
AT wortmanjeremyr externalcovid19deeplearningmodelvalidationonacrailabitsabravenewworld
AT medinaadam externalcovid19deeplearningmodelvalidationonacrailabitsabravenewworld
AT kalpathycramerjayashree externalcovid19deeplearningmodelvalidationonacrailabitsabravenewworld
AT waldchristoph externalcovid19deeplearningmodelvalidationonacrailabitsabravenewworld