Cargando…
Implementation and prospective real-time evaluation of a generalized system for in-clinic deployment and validation of machine learning models in radiology
The medical imaging community has embraced Machine Learning (ML) as evidenced by the rapid increase in the number of ML models being developed, but validating and deploying these models in the clinic remains a challenge. The engineering involved in integrating and assessing the efficacy of ML models...
Autores principales: | , , , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Public Library of Science
2023
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10441783/ https://www.ncbi.nlm.nih.gov/pubmed/37603542 http://dx.doi.org/10.1371/journal.pdig.0000227 |
_version_ | 1785093448530919424 |
---|---|
author | Hawkins, James R. Olson, Marram P. Harouni, Ahmed Qin, Ming Melvin Hess, Christopher P. Majumdar, Sharmila Crane, Jason C. |
author_facet | Hawkins, James R. Olson, Marram P. Harouni, Ahmed Qin, Ming Melvin Hess, Christopher P. Majumdar, Sharmila Crane, Jason C. |
author_sort | Hawkins, James R. |
collection | PubMed |
description | The medical imaging community has embraced Machine Learning (ML) as evidenced by the rapid increase in the number of ML models being developed, but validating and deploying these models in the clinic remains a challenge. The engineering involved in integrating and assessing the efficacy of ML models within the clinical workflow is complex. This paper presents a general-purpose, end-to-end, clinically integrated ML model deployment and validation system implemented at UCSF. Engineering and usability challenges and results from 3 use cases are presented. A generalized validation system based on free, open-source software (OSS) was implemented, connecting clinical imaging modalities, the Picture Archiving and Communication System (PACS), and an ML inference server. ML pipelines were implemented in NVIDIA’s Clara Deploy framework with results and clinician feedback stored in a customized XNAT instance, separate from the clinical record but linked from within PACS. Prospective clinical validation studies of 3 ML models were conducted, with data routed from multiple clinical imaging modalities and PACS. Completed validation studies provided expert clinical feedback on model performance and usability, plus system reliability and performance metrics. Clinical validation of ML models entails assessing model performance, impact on clinical infrastructure, robustness, and usability. Study results must be easily accessible to participating clinicians but remain outside the clinical record. Building a system that generalizes and scales across multiple ML models takes the concerted effort of software engineers, clinicians, data scientists, and system administrators, and benefits from the use of modular OSS. The present work provides a template for institutions looking to translate and clinically validate ML models in the clinic, together with required resources and expected challenges. |
format | Online Article Text |
id | pubmed-10441783 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2023 |
publisher | Public Library of Science |
record_format | MEDLINE/PubMed |
spelling | pubmed-104417832023-08-22 Implementation and prospective real-time evaluation of a generalized system for in-clinic deployment and validation of machine learning models in radiology Hawkins, James R. Olson, Marram P. Harouni, Ahmed Qin, Ming Melvin Hess, Christopher P. Majumdar, Sharmila Crane, Jason C. PLOS Digit Health Research Article The medical imaging community has embraced Machine Learning (ML) as evidenced by the rapid increase in the number of ML models being developed, but validating and deploying these models in the clinic remains a challenge. The engineering involved in integrating and assessing the efficacy of ML models within the clinical workflow is complex. This paper presents a general-purpose, end-to-end, clinically integrated ML model deployment and validation system implemented at UCSF. Engineering and usability challenges and results from 3 use cases are presented. A generalized validation system based on free, open-source software (OSS) was implemented, connecting clinical imaging modalities, the Picture Archiving and Communication System (PACS), and an ML inference server. ML pipelines were implemented in NVIDIA’s Clara Deploy framework with results and clinician feedback stored in a customized XNAT instance, separate from the clinical record but linked from within PACS. Prospective clinical validation studies of 3 ML models were conducted, with data routed from multiple clinical imaging modalities and PACS. Completed validation studies provided expert clinical feedback on model performance and usability, plus system reliability and performance metrics. Clinical validation of ML models entails assessing model performance, impact on clinical infrastructure, robustness, and usability. Study results must be easily accessible to participating clinicians but remain outside the clinical record. Building a system that generalizes and scales across multiple ML models takes the concerted effort of software engineers, clinicians, data scientists, and system administrators, and benefits from the use of modular OSS. The present work provides a template for institutions looking to translate and clinically validate ML models in the clinic, together with required resources and expected challenges. Public Library of Science 2023-08-21 /pmc/articles/PMC10441783/ /pubmed/37603542 http://dx.doi.org/10.1371/journal.pdig.0000227 Text en https://creativecommons.org/publicdomain/zero/1.0/This is an open access article, free of all copyright, and may be freely reproduced, distributed, transmitted, modified, built upon, or otherwise used by anyone for any lawful purpose. The work is made available under the Creative Commons CC0 (https://creativecommons.org/publicdomain/zero/1.0/) public domain dedication. |
spellingShingle | Research Article Hawkins, James R. Olson, Marram P. Harouni, Ahmed Qin, Ming Melvin Hess, Christopher P. Majumdar, Sharmila Crane, Jason C. Implementation and prospective real-time evaluation of a generalized system for in-clinic deployment and validation of machine learning models in radiology |
title | Implementation and prospective real-time evaluation of a generalized system for in-clinic deployment and validation of machine learning models in radiology |
title_full | Implementation and prospective real-time evaluation of a generalized system for in-clinic deployment and validation of machine learning models in radiology |
title_fullStr | Implementation and prospective real-time evaluation of a generalized system for in-clinic deployment and validation of machine learning models in radiology |
title_full_unstemmed | Implementation and prospective real-time evaluation of a generalized system for in-clinic deployment and validation of machine learning models in radiology |
title_short | Implementation and prospective real-time evaluation of a generalized system for in-clinic deployment and validation of machine learning models in radiology |
title_sort | implementation and prospective real-time evaluation of a generalized system for in-clinic deployment and validation of machine learning models in radiology |
topic | Research Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10441783/ https://www.ncbi.nlm.nih.gov/pubmed/37603542 http://dx.doi.org/10.1371/journal.pdig.0000227 |
work_keys_str_mv | AT hawkinsjamesr implementationandprospectiverealtimeevaluationofageneralizedsystemforinclinicdeploymentandvalidationofmachinelearningmodelsinradiology AT olsonmarramp implementationandprospectiverealtimeevaluationofageneralizedsystemforinclinicdeploymentandvalidationofmachinelearningmodelsinradiology AT harouniahmed implementationandprospectiverealtimeevaluationofageneralizedsystemforinclinicdeploymentandvalidationofmachinelearningmodelsinradiology AT qinmingmelvin implementationandprospectiverealtimeevaluationofageneralizedsystemforinclinicdeploymentandvalidationofmachinelearningmodelsinradiology AT hesschristopherp implementationandprospectiverealtimeevaluationofageneralizedsystemforinclinicdeploymentandvalidationofmachinelearningmodelsinradiology AT majumdarsharmila implementationandprospectiverealtimeevaluationofageneralizedsystemforinclinicdeploymentandvalidationofmachinelearningmodelsinradiology AT cranejasonc implementationandprospectiverealtimeevaluationofageneralizedsystemforinclinicdeploymentandvalidationofmachinelearningmodelsinradiology |