Multicenter Validation of Deep Learning Algorithm ROP.AI for the Automated Diagnosis of Plus Disease in ROP

PURPOSE: Retinopathy of prematurity (ROP) is a sight-threatening vasoproliferative retinal disease affecting premature infants. The detection of plus disease, a severe form of ROP requiring treatment, remains challenging owing to subjectivity, frequency, and time intensity of retinal examinations. R...

Descripción completa

Detalles Bibliográficos
Autores principales: Bai, Amelia, Dai, Shuan, Hung, Jacky, Kirpalani, Aditi, Russell, Heather, Elder, James, Shah, Shaheen, Carty, Christopher, Tan, Zachary
Formato: Online Artículo Texto
Lenguaje:English
Publicado: The Association for Research in Vision and Ophthalmology 2023
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10431208/
https://www.ncbi.nlm.nih.gov/pubmed/37578427
http://dx.doi.org/10.1167/tvst.12.8.13
_version_ 1785091146745118720
author Bai, Amelia
Dai, Shuan
Hung, Jacky
Kirpalani, Aditi
Russell, Heather
Elder, James
Shah, Shaheen
Carty, Christopher
Tan, Zachary
author_facet Bai, Amelia
Dai, Shuan
Hung, Jacky
Kirpalani, Aditi
Russell, Heather
Elder, James
Shah, Shaheen
Carty, Christopher
Tan, Zachary
author_sort Bai, Amelia
collection PubMed
description PURPOSE: Retinopathy of prematurity (ROP) is a sight-threatening vasoproliferative retinal disease affecting premature infants. The detection of plus disease, a severe form of ROP requiring treatment, remains challenging owing to subjectivity, frequency, and time intensity of retinal examinations. Recent artificial intelligence (AI) algorithms developed to detect plus disease aims to alleviate these challenges; however, they have not been tested against a diverse neonatal population. Our study aims to validate ROP.AI, an AI algorithm developed from a single cohort, against a multicenter Australian cohort to determine its performance in detecting plus disease. METHODS: Retinal images captured during routine ROP screening from May 2021 to February 2022 across five major tertiary centers throughout Australia were collected and uploaded to ROP.AI. AI diagnostic output was compared with one of five ROP experts. Sensitivity, specificity, negative predictive value, and area under the receiver operator curve were determined. RESULTS: We collected 8052 images. The area under the receiver operator curve for the diagnosis of plus disease was 0.75. ROP.AI achieved 84% sensitivity, 43% specificity, and 96% negative predictive value for the detection of plus disease after operating point optimization. CONCLUSIONS: ROP.AI was able to detect plus disease in an external, multicenter cohort despite being trained from a single center. Algorithm performance was demonstrated without preprocessing or augmentation, simulating real-world clinical applicability. Further training may improve generalizability for clinical implementation. TRANSLATIONAL RELEVANCE: These results demonstrate ROP.AI's potential as a screening tool for the detection of plus disease in future clinical practice and provides a solution to overcome current diagnostic challenges.
format Online
Article
Text
id pubmed-10431208
institution National Center for Biotechnology Information
language English
publishDate 2023
publisher The Association for Research in Vision and Ophthalmology
record_format MEDLINE/PubMed
spelling pubmed-104312082023-08-17 Multicenter Validation of Deep Learning Algorithm ROP.AI for the Automated Diagnosis of Plus Disease in ROP Bai, Amelia Dai, Shuan Hung, Jacky Kirpalani, Aditi Russell, Heather Elder, James Shah, Shaheen Carty, Christopher Tan, Zachary Transl Vis Sci Technol Artificial Intelligence PURPOSE: Retinopathy of prematurity (ROP) is a sight-threatening vasoproliferative retinal disease affecting premature infants. The detection of plus disease, a severe form of ROP requiring treatment, remains challenging owing to subjectivity, frequency, and time intensity of retinal examinations. Recent artificial intelligence (AI) algorithms developed to detect plus disease aims to alleviate these challenges; however, they have not been tested against a diverse neonatal population. Our study aims to validate ROP.AI, an AI algorithm developed from a single cohort, against a multicenter Australian cohort to determine its performance in detecting plus disease. METHODS: Retinal images captured during routine ROP screening from May 2021 to February 2022 across five major tertiary centers throughout Australia were collected and uploaded to ROP.AI. AI diagnostic output was compared with one of five ROP experts. Sensitivity, specificity, negative predictive value, and area under the receiver operator curve were determined. RESULTS: We collected 8052 images. The area under the receiver operator curve for the diagnosis of plus disease was 0.75. ROP.AI achieved 84% sensitivity, 43% specificity, and 96% negative predictive value for the detection of plus disease after operating point optimization. CONCLUSIONS: ROP.AI was able to detect plus disease in an external, multicenter cohort despite being trained from a single center. Algorithm performance was demonstrated without preprocessing or augmentation, simulating real-world clinical applicability. Further training may improve generalizability for clinical implementation. TRANSLATIONAL RELEVANCE: These results demonstrate ROP.AI's potential as a screening tool for the detection of plus disease in future clinical practice and provides a solution to overcome current diagnostic challenges. The Association for Research in Vision and Ophthalmology 2023-08-14 /pmc/articles/PMC10431208/ /pubmed/37578427 http://dx.doi.org/10.1167/tvst.12.8.13 Text en Copyright 2023 The Authors https://creativecommons.org/licenses/by-nc-nd/4.0/This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
spellingShingle Artificial Intelligence
Bai, Amelia
Dai, Shuan
Hung, Jacky
Kirpalani, Aditi
Russell, Heather
Elder, James
Shah, Shaheen
Carty, Christopher
Tan, Zachary
Multicenter Validation of Deep Learning Algorithm ROP.AI for the Automated Diagnosis of Plus Disease in ROP
title Multicenter Validation of Deep Learning Algorithm ROP.AI for the Automated Diagnosis of Plus Disease in ROP
title_full Multicenter Validation of Deep Learning Algorithm ROP.AI for the Automated Diagnosis of Plus Disease in ROP
title_fullStr Multicenter Validation of Deep Learning Algorithm ROP.AI for the Automated Diagnosis of Plus Disease in ROP
title_full_unstemmed Multicenter Validation of Deep Learning Algorithm ROP.AI for the Automated Diagnosis of Plus Disease in ROP
title_short Multicenter Validation of Deep Learning Algorithm ROP.AI for the Automated Diagnosis of Plus Disease in ROP
title_sort multicenter validation of deep learning algorithm rop.ai for the automated diagnosis of plus disease in rop
topic Artificial Intelligence
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10431208/
https://www.ncbi.nlm.nih.gov/pubmed/37578427
http://dx.doi.org/10.1167/tvst.12.8.13
work_keys_str_mv AT baiamelia multicentervalidationofdeeplearningalgorithmropaifortheautomateddiagnosisofplusdiseaseinrop
AT daishuan multicentervalidationofdeeplearningalgorithmropaifortheautomateddiagnosisofplusdiseaseinrop
AT hungjacky multicentervalidationofdeeplearningalgorithmropaifortheautomateddiagnosisofplusdiseaseinrop
AT kirpalaniaditi multicentervalidationofdeeplearningalgorithmropaifortheautomateddiagnosisofplusdiseaseinrop
AT russellheather multicentervalidationofdeeplearningalgorithmropaifortheautomateddiagnosisofplusdiseaseinrop
AT elderjames multicentervalidationofdeeplearningalgorithmropaifortheautomateddiagnosisofplusdiseaseinrop
AT shahshaheen multicentervalidationofdeeplearningalgorithmropaifortheautomateddiagnosisofplusdiseaseinrop
AT cartychristopher multicentervalidationofdeeplearningalgorithmropaifortheautomateddiagnosisofplusdiseaseinrop
AT tanzachary multicentervalidationofdeeplearningalgorithmropaifortheautomateddiagnosisofplusdiseaseinrop