Cargando…
Accelerating voxelwise annotation of cross-sectional imaging through AI collaborative labeling with quality assurance and bias mitigation
BACKGROUND: precision-medicine quantitative tools for cross-sectional imaging require painstaking labeling of targets that vary considerably in volume, prohibiting scaling of data annotation efforts and supervised training to large datasets for robust and generalizable clinical performance. A straig...
Autores principales: | , , , , , , , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Frontiers Media S.A.
2023
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10362988/ https://www.ncbi.nlm.nih.gov/pubmed/37485306 http://dx.doi.org/10.3389/fradi.2023.1202412 |
_version_ | 1785076544634355712 |
---|---|
author | Dreizin, David Zhang, Lei Sarkar, Nathan Bodanapally, Uttam K. Li, Guang Hu, Jiazhen Chen, Haomin Khedr, Mustafa Khetan, Udit Campbell, Peter Unberath, Mathias |
author_facet | Dreizin, David Zhang, Lei Sarkar, Nathan Bodanapally, Uttam K. Li, Guang Hu, Jiazhen Chen, Haomin Khedr, Mustafa Khetan, Udit Campbell, Peter Unberath, Mathias |
author_sort | Dreizin, David |
collection | PubMed |
description | BACKGROUND: precision-medicine quantitative tools for cross-sectional imaging require painstaking labeling of targets that vary considerably in volume, prohibiting scaling of data annotation efforts and supervised training to large datasets for robust and generalizable clinical performance. A straight-forward time-saving strategy involves manual editing of AI-generated labels, which we call AI-collaborative labeling (AICL). Factors affecting the efficacy and utility of such an approach are unknown. Reduction in time effort is not well documented. Further, edited AI labels may be prone to automation bias. PURPOSE: In this pilot, using a cohort of CTs with intracavitary hemorrhage, we evaluate both time savings and AICL label quality and propose criteria that must be met for using AICL annotations as a high-throughput, high-quality ground truth. METHODS: 57 CT scans of patients with traumatic intracavitary hemorrhage were included. No participant recruited for this study had previously interpreted the scans. nnU-net models trained on small existing datasets for each feature (hemothorax/hemoperitoneum/pelvic hematoma; n = 77–253) were used in inference. Two common scenarios served as baseline comparison- de novo expert manual labeling, and expert edits of trained staff labels. Parameters included time effort and image quality graded by a blinded independent expert using a 9-point scale. The observer also attempted to discriminate AICL and expert labels in a random subset (n = 18). Data were compared with ANOVA and post-hoc paired signed rank tests with Bonferroni correction. RESULTS: AICL reduced time effort 2.8-fold compared to staff label editing, and 8.7-fold compared to expert labeling (corrected p < 0.0006). Mean Likert grades for AICL (8.4, SD:0.6) were significantly higher than for expert labels (7.8, SD:0.9) and edited staff labels (7.7, SD:0.8) (corrected p < 0.0006). The independent observer failed to correctly discriminate AI and human labels. CONCLUSION: For our use case and annotators, AICL facilitates rapid large-scale curation of high-quality ground truth. The proposed quality control regime can be employed by other investigators prior to embarking on AICL for segmentation tasks in large datasets. |
format | Online Article Text |
id | pubmed-10362988 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2023 |
publisher | Frontiers Media S.A. |
record_format | MEDLINE/PubMed |
spelling | pubmed-103629882023-07-22 Accelerating voxelwise annotation of cross-sectional imaging through AI collaborative labeling with quality assurance and bias mitigation Dreizin, David Zhang, Lei Sarkar, Nathan Bodanapally, Uttam K. Li, Guang Hu, Jiazhen Chen, Haomin Khedr, Mustafa Khetan, Udit Campbell, Peter Unberath, Mathias Front Radiol Radiology BACKGROUND: precision-medicine quantitative tools for cross-sectional imaging require painstaking labeling of targets that vary considerably in volume, prohibiting scaling of data annotation efforts and supervised training to large datasets for robust and generalizable clinical performance. A straight-forward time-saving strategy involves manual editing of AI-generated labels, which we call AI-collaborative labeling (AICL). Factors affecting the efficacy and utility of such an approach are unknown. Reduction in time effort is not well documented. Further, edited AI labels may be prone to automation bias. PURPOSE: In this pilot, using a cohort of CTs with intracavitary hemorrhage, we evaluate both time savings and AICL label quality and propose criteria that must be met for using AICL annotations as a high-throughput, high-quality ground truth. METHODS: 57 CT scans of patients with traumatic intracavitary hemorrhage were included. No participant recruited for this study had previously interpreted the scans. nnU-net models trained on small existing datasets for each feature (hemothorax/hemoperitoneum/pelvic hematoma; n = 77–253) were used in inference. Two common scenarios served as baseline comparison- de novo expert manual labeling, and expert edits of trained staff labels. Parameters included time effort and image quality graded by a blinded independent expert using a 9-point scale. The observer also attempted to discriminate AICL and expert labels in a random subset (n = 18). Data were compared with ANOVA and post-hoc paired signed rank tests with Bonferroni correction. RESULTS: AICL reduced time effort 2.8-fold compared to staff label editing, and 8.7-fold compared to expert labeling (corrected p < 0.0006). Mean Likert grades for AICL (8.4, SD:0.6) were significantly higher than for expert labels (7.8, SD:0.9) and edited staff labels (7.7, SD:0.8) (corrected p < 0.0006). The independent observer failed to correctly discriminate AI and human labels. CONCLUSION: For our use case and annotators, AICL facilitates rapid large-scale curation of high-quality ground truth. The proposed quality control regime can be employed by other investigators prior to embarking on AICL for segmentation tasks in large datasets. Frontiers Media S.A. 2023-07-11 /pmc/articles/PMC10362988/ /pubmed/37485306 http://dx.doi.org/10.3389/fradi.2023.1202412 Text en © 2023 Dreizin, Zhang, Sarkar, Bodanapally, Li, Hu, Chen, Khedr, Khetan, Campbell and Unberath. https://creativecommons.org/licenses/by/4.0/This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) (https://creativecommons.org/licenses/by/4.0/) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms. |
spellingShingle | Radiology Dreizin, David Zhang, Lei Sarkar, Nathan Bodanapally, Uttam K. Li, Guang Hu, Jiazhen Chen, Haomin Khedr, Mustafa Khetan, Udit Campbell, Peter Unberath, Mathias Accelerating voxelwise annotation of cross-sectional imaging through AI collaborative labeling with quality assurance and bias mitigation |
title | Accelerating voxelwise annotation of cross-sectional imaging through AI collaborative labeling with quality assurance and bias mitigation |
title_full | Accelerating voxelwise annotation of cross-sectional imaging through AI collaborative labeling with quality assurance and bias mitigation |
title_fullStr | Accelerating voxelwise annotation of cross-sectional imaging through AI collaborative labeling with quality assurance and bias mitigation |
title_full_unstemmed | Accelerating voxelwise annotation of cross-sectional imaging through AI collaborative labeling with quality assurance and bias mitigation |
title_short | Accelerating voxelwise annotation of cross-sectional imaging through AI collaborative labeling with quality assurance and bias mitigation |
title_sort | accelerating voxelwise annotation of cross-sectional imaging through ai collaborative labeling with quality assurance and bias mitigation |
topic | Radiology |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10362988/ https://www.ncbi.nlm.nih.gov/pubmed/37485306 http://dx.doi.org/10.3389/fradi.2023.1202412 |
work_keys_str_mv | AT dreizindavid acceleratingvoxelwiseannotationofcrosssectionalimagingthroughaicollaborativelabelingwithqualityassuranceandbiasmitigation AT zhanglei acceleratingvoxelwiseannotationofcrosssectionalimagingthroughaicollaborativelabelingwithqualityassuranceandbiasmitigation AT sarkarnathan acceleratingvoxelwiseannotationofcrosssectionalimagingthroughaicollaborativelabelingwithqualityassuranceandbiasmitigation AT bodanapallyuttamk acceleratingvoxelwiseannotationofcrosssectionalimagingthroughaicollaborativelabelingwithqualityassuranceandbiasmitigation AT liguang acceleratingvoxelwiseannotationofcrosssectionalimagingthroughaicollaborativelabelingwithqualityassuranceandbiasmitigation AT hujiazhen acceleratingvoxelwiseannotationofcrosssectionalimagingthroughaicollaborativelabelingwithqualityassuranceandbiasmitigation AT chenhaomin acceleratingvoxelwiseannotationofcrosssectionalimagingthroughaicollaborativelabelingwithqualityassuranceandbiasmitigation AT khedrmustafa acceleratingvoxelwiseannotationofcrosssectionalimagingthroughaicollaborativelabelingwithqualityassuranceandbiasmitigation AT khetanudit acceleratingvoxelwiseannotationofcrosssectionalimagingthroughaicollaborativelabelingwithqualityassuranceandbiasmitigation AT campbellpeter acceleratingvoxelwiseannotationofcrosssectionalimagingthroughaicollaborativelabelingwithqualityassuranceandbiasmitigation AT unberathmathias acceleratingvoxelwiseannotationofcrosssectionalimagingthroughaicollaborativelabelingwithqualityassuranceandbiasmitigation |