Cargando…

Studying human-AI collaboration protocols: the case of the Kasparov’s law in radiological double reading

PURPOSE: The integration of Artificial Intelligence into medical practices has recently been advocated for the promise to bring increased efficiency and effectiveness to these practices. Nonetheless, little research has so far been aimed at understanding the best human-AI interaction protocols in co...

Descripción completa

Detalles Bibliográficos
Autores principales: Cabitza, Federico, Campagner, Andrea, Sconfienza, Luca Maria
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Springer International Publishing 2021
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7864624/
https://www.ncbi.nlm.nih.gov/pubmed/33585029
http://dx.doi.org/10.1007/s13755-021-00138-8
_version_ 1783647692999622656
author Cabitza, Federico
Campagner, Andrea
Sconfienza, Luca Maria
author_facet Cabitza, Federico
Campagner, Andrea
Sconfienza, Luca Maria
author_sort Cabitza, Federico
collection PubMed
description PURPOSE: The integration of Artificial Intelligence into medical practices has recently been advocated for the promise to bring increased efficiency and effectiveness to these practices. Nonetheless, little research has so far been aimed at understanding the best human-AI interaction protocols in collaborative tasks, even in currently more viable settings, like independent double-reading screening tasks. METHODS: To this aim, we report about a retrospective case–control study, involving 12 board-certified radiologists, in the detection of knee lesions by means of Magnetic Resonance Imaging, in which we simulated the serial combination of two Deep Learning models with humans in eight double-reading protocols. Inspired by the so-called Kasparov’s Laws, we investigate whether the combination of humans and AI models could achieve better performance than AI models alone, and whether weak reader, when supported by fit-for-use interaction protocols, could out-perform stronger readers. RESULTS: We discuss two main findings: groups of humans who perform significantly worse than a state-of-the-art AI can significantly outperform it if their judgements are aggregated by majority voting (in concordance with the first part of the Kasparov’s law); small ensembles of significantly weaker readers can significantly outperform teams of stronger readers, supported by the same computational tool, when the judgments of the former ones are combined within “fit-for-use” protocols (in concordance with the second part of the Kasparov’s law). CONCLUSION: Our study shows that good interaction protocols can guarantee improved decision performance that easily surpasses the performance of individual agents, even of realistic super-human AI systems. This finding highlights the importance of focusing on how to guarantee better co-operation within human-AI teams, so to enable safer and more human sustainable care practices.
format Online
Article
Text
id pubmed-7864624
institution National Center for Biotechnology Information
language English
publishDate 2021
publisher Springer International Publishing
record_format MEDLINE/PubMed
spelling pubmed-78646242021-02-09 Studying human-AI collaboration protocols: the case of the Kasparov’s law in radiological double reading Cabitza, Federico Campagner, Andrea Sconfienza, Luca Maria Health Inf Sci Syst Research PURPOSE: The integration of Artificial Intelligence into medical practices has recently been advocated for the promise to bring increased efficiency and effectiveness to these practices. Nonetheless, little research has so far been aimed at understanding the best human-AI interaction protocols in collaborative tasks, even in currently more viable settings, like independent double-reading screening tasks. METHODS: To this aim, we report about a retrospective case–control study, involving 12 board-certified radiologists, in the detection of knee lesions by means of Magnetic Resonance Imaging, in which we simulated the serial combination of two Deep Learning models with humans in eight double-reading protocols. Inspired by the so-called Kasparov’s Laws, we investigate whether the combination of humans and AI models could achieve better performance than AI models alone, and whether weak reader, when supported by fit-for-use interaction protocols, could out-perform stronger readers. RESULTS: We discuss two main findings: groups of humans who perform significantly worse than a state-of-the-art AI can significantly outperform it if their judgements are aggregated by majority voting (in concordance with the first part of the Kasparov’s law); small ensembles of significantly weaker readers can significantly outperform teams of stronger readers, supported by the same computational tool, when the judgments of the former ones are combined within “fit-for-use” protocols (in concordance with the second part of the Kasparov’s law). CONCLUSION: Our study shows that good interaction protocols can guarantee improved decision performance that easily surpasses the performance of individual agents, even of realistic super-human AI systems. This finding highlights the importance of focusing on how to guarantee better co-operation within human-AI teams, so to enable safer and more human sustainable care practices. Springer International Publishing 2021-02-05 /pmc/articles/PMC7864624/ /pubmed/33585029 http://dx.doi.org/10.1007/s13755-021-00138-8 Text en © The Author(s) 2021 https://creativecommons.org/licenses/by/4.0/Open AccessThis article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ (https://creativecommons.org/licenses/by/4.0/) .
spellingShingle Research
Cabitza, Federico
Campagner, Andrea
Sconfienza, Luca Maria
Studying human-AI collaboration protocols: the case of the Kasparov’s law in radiological double reading
title Studying human-AI collaboration protocols: the case of the Kasparov’s law in radiological double reading
title_full Studying human-AI collaboration protocols: the case of the Kasparov’s law in radiological double reading
title_fullStr Studying human-AI collaboration protocols: the case of the Kasparov’s law in radiological double reading
title_full_unstemmed Studying human-AI collaboration protocols: the case of the Kasparov’s law in radiological double reading
title_short Studying human-AI collaboration protocols: the case of the Kasparov’s law in radiological double reading
title_sort studying human-ai collaboration protocols: the case of the kasparov’s law in radiological double reading
topic Research
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7864624/
https://www.ncbi.nlm.nih.gov/pubmed/33585029
http://dx.doi.org/10.1007/s13755-021-00138-8
work_keys_str_mv AT cabitzafederico studyinghumanaicollaborationprotocolsthecaseofthekasparovslawinradiologicaldoublereading
AT campagnerandrea studyinghumanaicollaborationprotocolsthecaseofthekasparovslawinradiologicaldoublereading
AT sconfienzalucamaria studyinghumanaicollaborationprotocolsthecaseofthekasparovslawinradiologicaldoublereading