Evaluation of physician performance using a concurrent-read artificial intelligence system to support breast ultrasound interpretation

PURPOSE: The purpose of this study was to compare the diagnostic performance and the interpretation time of breast ultrasound examination between reading without and with the artificial intelligence (AI) system as a concurrent reading aid. MATERIAL AND METHODS: A fully crossed multi-reader and multi-case (MRMC) reader study was conducted. Sixteen participating physicians were recruited and retrospectively interpreted 172 breast ultrasound cases in two reading scenarios, once without and once with the AI system (BU-CAD™, TaiHao Medical Inc.) assistance for concurrent reading. Interpretations of any given case set with and without the AI system were separated by at least 5 weeks. These reading results were compared to the reference standard and the area under the LROC curve (AUCLROC), sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated for performance evaluations. The interpretation time was also compared between the unaided and aided scenarios. RESULTS: With the help of the AI system, the readers had higher diagnostic performance with an increase in the average AUCLROC from 0.7582 to 0.8294 with statistically significant. The sensitivity, specificity, PPV, and NPV were also improved from 95.77%, 24.07%, 44.18%, and 93.50%–98.17%, 30.67%, 46.91%, and 96.10%, respectively. Of these, the improvement in specificity reached statistical significance. The average interpretation time was significantly reduced by approximately 40% when the readers were assisted by the AI system. CONCLUSION: The concurrent-read AI system improves the diagnostic performance in detecting and diagnosing breast lesions on breast ultrasound images. In addition, the interpretation time is effectively reduced for the interpreting physicians.