Cargando…

Evaluation framework to guide implementation of AI systems into healthcare settings

OBJECTIVES: To date, many artificial intelligence (AI) systems have been developed in healthcare, but adoption has been limited. This may be due to inappropriate or incomplete evaluation and a lack of internationally recognised AI standards on evaluation. To have confidence in the generalisability o...

Descripción completa

Detalles Bibliográficos
Autores principales: Reddy, Sandeep, Rogers, Wendy, Makinen, Ville-Petteri, Coiera, Enrico, Brown, Pieta, Wenzel, Markus, Weicken, Eva, Ansari, Saba, Mathur, Piyush, Casey, Aaron, Kelly, Blair
Formato: Online Artículo Texto
Lenguaje:English
Publicado: BMJ Publishing Group 2021
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8513218/
https://www.ncbi.nlm.nih.gov/pubmed/34642177
http://dx.doi.org/10.1136/bmjhci-2021-100444
_version_ 1784583170042101760
author Reddy, Sandeep
Rogers, Wendy
Makinen, Ville-Petteri
Coiera, Enrico
Brown, Pieta
Wenzel, Markus
Weicken, Eva
Ansari, Saba
Mathur, Piyush
Casey, Aaron
Kelly, Blair
author_facet Reddy, Sandeep
Rogers, Wendy
Makinen, Ville-Petteri
Coiera, Enrico
Brown, Pieta
Wenzel, Markus
Weicken, Eva
Ansari, Saba
Mathur, Piyush
Casey, Aaron
Kelly, Blair
author_sort Reddy, Sandeep
collection PubMed
description OBJECTIVES: To date, many artificial intelligence (AI) systems have been developed in healthcare, but adoption has been limited. This may be due to inappropriate or incomplete evaluation and a lack of internationally recognised AI standards on evaluation. To have confidence in the generalisability of AI systems in healthcare and to enable their integration into workflows, there is a need for a practical yet comprehensive instrument to assess the translational aspects of the available AI systems. Currently available evaluation frameworks for AI in healthcare focus on the reporting and regulatory aspects but have little guidance regarding assessment of the translational aspects of the AI systems like the functional, utility and ethical components. METHODS: To address this gap and create a framework that assesses real-world systems, an international team has developed a translationally focused evaluation framework termed ‘Translational Evaluation of Healthcare AI (TEHAI)’. A critical review of literature assessed existing evaluation and reporting frameworks and gaps. Next, using health technology evaluation and translational principles, reporting components were identified for consideration. These were independently reviewed for consensus inclusion in a final framework by an international panel of eight expert. RESULTS: TEHAI includes three main components: capability, utility and adoption. The emphasis on translational and ethical features of the model development and deployment distinguishes TEHAI from other evaluation instruments. In specific, the evaluation components can be applied at any stage of the development and deployment of the AI system. DISCUSSION: One major limitation of existing reporting or evaluation frameworks is their narrow focus. TEHAI, because of its strong foundation in translation research models and an emphasis on safety, translational value and generalisability, not only has a theoretical basis but also practical application to assessing real-world systems. CONCLUSION: The translational research theoretic approach used to develop TEHAI should see it having application not just for evaluation of clinical AI in research settings, but more broadly to guide evaluation of working clinical systems.
format Online
Article
Text
id pubmed-8513218
institution National Center for Biotechnology Information
language English
publishDate 2021
publisher BMJ Publishing Group
record_format MEDLINE/PubMed
spelling pubmed-85132182021-10-27 Evaluation framework to guide implementation of AI systems into healthcare settings Reddy, Sandeep Rogers, Wendy Makinen, Ville-Petteri Coiera, Enrico Brown, Pieta Wenzel, Markus Weicken, Eva Ansari, Saba Mathur, Piyush Casey, Aaron Kelly, Blair BMJ Health Care Inform Review OBJECTIVES: To date, many artificial intelligence (AI) systems have been developed in healthcare, but adoption has been limited. This may be due to inappropriate or incomplete evaluation and a lack of internationally recognised AI standards on evaluation. To have confidence in the generalisability of AI systems in healthcare and to enable their integration into workflows, there is a need for a practical yet comprehensive instrument to assess the translational aspects of the available AI systems. Currently available evaluation frameworks for AI in healthcare focus on the reporting and regulatory aspects but have little guidance regarding assessment of the translational aspects of the AI systems like the functional, utility and ethical components. METHODS: To address this gap and create a framework that assesses real-world systems, an international team has developed a translationally focused evaluation framework termed ‘Translational Evaluation of Healthcare AI (TEHAI)’. A critical review of literature assessed existing evaluation and reporting frameworks and gaps. Next, using health technology evaluation and translational principles, reporting components were identified for consideration. These were independently reviewed for consensus inclusion in a final framework by an international panel of eight expert. RESULTS: TEHAI includes three main components: capability, utility and adoption. The emphasis on translational and ethical features of the model development and deployment distinguishes TEHAI from other evaluation instruments. In specific, the evaluation components can be applied at any stage of the development and deployment of the AI system. DISCUSSION: One major limitation of existing reporting or evaluation frameworks is their narrow focus. TEHAI, because of its strong foundation in translation research models and an emphasis on safety, translational value and generalisability, not only has a theoretical basis but also practical application to assessing real-world systems. CONCLUSION: The translational research theoretic approach used to develop TEHAI should see it having application not just for evaluation of clinical AI in research settings, but more broadly to guide evaluation of working clinical systems. BMJ Publishing Group 2021-10-12 /pmc/articles/PMC8513218/ /pubmed/34642177 http://dx.doi.org/10.1136/bmjhci-2021-100444 Text en © Author(s) (or their employer(s)) 2021. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ. https://creativecommons.org/licenses/by-nc/4.0/This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/ (https://creativecommons.org/licenses/by-nc/4.0/) .
spellingShingle Review
Reddy, Sandeep
Rogers, Wendy
Makinen, Ville-Petteri
Coiera, Enrico
Brown, Pieta
Wenzel, Markus
Weicken, Eva
Ansari, Saba
Mathur, Piyush
Casey, Aaron
Kelly, Blair
Evaluation framework to guide implementation of AI systems into healthcare settings
title Evaluation framework to guide implementation of AI systems into healthcare settings
title_full Evaluation framework to guide implementation of AI systems into healthcare settings
title_fullStr Evaluation framework to guide implementation of AI systems into healthcare settings
title_full_unstemmed Evaluation framework to guide implementation of AI systems into healthcare settings
title_short Evaluation framework to guide implementation of AI systems into healthcare settings
title_sort evaluation framework to guide implementation of ai systems into healthcare settings
topic Review
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8513218/
https://www.ncbi.nlm.nih.gov/pubmed/34642177
http://dx.doi.org/10.1136/bmjhci-2021-100444
work_keys_str_mv AT reddysandeep evaluationframeworktoguideimplementationofaisystemsintohealthcaresettings
AT rogerswendy evaluationframeworktoguideimplementationofaisystemsintohealthcaresettings
AT makinenvillepetteri evaluationframeworktoguideimplementationofaisystemsintohealthcaresettings
AT coieraenrico evaluationframeworktoguideimplementationofaisystemsintohealthcaresettings
AT brownpieta evaluationframeworktoguideimplementationofaisystemsintohealthcaresettings
AT wenzelmarkus evaluationframeworktoguideimplementationofaisystemsintohealthcaresettings
AT weickeneva evaluationframeworktoguideimplementationofaisystemsintohealthcaresettings
AT ansarisaba evaluationframeworktoguideimplementationofaisystemsintohealthcaresettings
AT mathurpiyush evaluationframeworktoguideimplementationofaisystemsintohealthcaresettings
AT caseyaaron evaluationframeworktoguideimplementationofaisystemsintohealthcaresettings
AT kellyblair evaluationframeworktoguideimplementationofaisystemsintohealthcaresettings