Cargando…
Developing the surgeon-machine interface: using a novel instance-segmentation framework for intraoperative landmark labelling
INTRODUCTION: The utilisation of artificial intelligence (AI) augments intraoperative safety, surgical training, and patient outcomes. We introduce the term Surgeon-Machine Interface (SMI) to describe this innovative intersection between surgeons and machine inference. A custom deep computer vision...
Autores principales: | , , , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Frontiers Media S.A.
2023
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10626480/ https://www.ncbi.nlm.nih.gov/pubmed/37936949 http://dx.doi.org/10.3389/fsurg.2023.1259756 |
_version_ | 1785131345618403328 |
---|---|
author | Park, Jay J. Doiphode, Nehal Zhang, Xiao Pan, Lishuo Blue, Rachel Shi, Jianbo Buch, Vivek P. |
author_facet | Park, Jay J. Doiphode, Nehal Zhang, Xiao Pan, Lishuo Blue, Rachel Shi, Jianbo Buch, Vivek P. |
author_sort | Park, Jay J. |
collection | PubMed |
description | INTRODUCTION: The utilisation of artificial intelligence (AI) augments intraoperative safety, surgical training, and patient outcomes. We introduce the term Surgeon-Machine Interface (SMI) to describe this innovative intersection between surgeons and machine inference. A custom deep computer vision (CV) architecture within a sparse labelling paradigm was developed, specifically tailored to conceptualise the SMI. This platform demonstrates the ability to perform instance segmentation on anatomical landmarks and tools from a single open spinal dural arteriovenous fistula (dAVF) surgery video dataset. METHODS: Our custom deep convolutional neural network was based on SOLOv2 architecture for precise, instance-level segmentation of surgical video data. Test video consisted of 8520 frames, with sparse labelling of only 133 frames annotated for training. Accuracy and inference time, assessed using F1-score and mean Average Precision (mAP), were compared against current state-of-the-art architectures on a separate test set of 85 additionally annotated frames. RESULTS: Our SMI demonstrated superior accuracy and computing speed compared to these frameworks. The F1-score and mAP achieved by our platform were 17% and 15.2% respectively, surpassing MaskRCNN (15.2%, 13.9%), YOLOv3 (5.4%, 11.9%), and SOLOv2 (3.1%, 10.4%). Considering detections that exceeded the Intersection over Union threshold of 50%, our platform achieved an impressive F1-score of 44.2% and mAP of 46.3%, outperforming MaskRCNN (41.3%, 43.5%), YOLOv3 (15%, 34.1%), and SOLOv2 (9%, 32.3%). Our platform demonstrated the fastest inference time (88ms), compared to MaskRCNN (90ms), SOLOV2 (100ms), and YOLOv3 (106ms). Finally, the minimal amount of training set demonstrated a good generalisation performance –our architecture successfully identified objects in a frame that were not included in the training or validation frames, indicating its ability to handle out-of-domain scenarios. DISCUSSION: We present our development of an innovative intraoperative SMI to demonstrate the future promise of advanced CV in the surgical domain. Through successful implementation in a microscopic dAVF surgery, our framework demonstrates superior performance over current state-of-the-art segmentation architectures in intraoperative landmark guidance with high sample efficiency, representing the most advanced AI-enabled surgical inference platform to date. Our future goals include transfer learning paradigms for scaling to additional surgery types, addressing clinical and technical limitations for performing real-time decoding, and ultimate enablement of a real-time neurosurgical guidance platform. |
format | Online Article Text |
id | pubmed-10626480 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2023 |
publisher | Frontiers Media S.A. |
record_format | MEDLINE/PubMed |
spelling | pubmed-106264802023-11-07 Developing the surgeon-machine interface: using a novel instance-segmentation framework for intraoperative landmark labelling Park, Jay J. Doiphode, Nehal Zhang, Xiao Pan, Lishuo Blue, Rachel Shi, Jianbo Buch, Vivek P. Front Surg Surgery INTRODUCTION: The utilisation of artificial intelligence (AI) augments intraoperative safety, surgical training, and patient outcomes. We introduce the term Surgeon-Machine Interface (SMI) to describe this innovative intersection between surgeons and machine inference. A custom deep computer vision (CV) architecture within a sparse labelling paradigm was developed, specifically tailored to conceptualise the SMI. This platform demonstrates the ability to perform instance segmentation on anatomical landmarks and tools from a single open spinal dural arteriovenous fistula (dAVF) surgery video dataset. METHODS: Our custom deep convolutional neural network was based on SOLOv2 architecture for precise, instance-level segmentation of surgical video data. Test video consisted of 8520 frames, with sparse labelling of only 133 frames annotated for training. Accuracy and inference time, assessed using F1-score and mean Average Precision (mAP), were compared against current state-of-the-art architectures on a separate test set of 85 additionally annotated frames. RESULTS: Our SMI demonstrated superior accuracy and computing speed compared to these frameworks. The F1-score and mAP achieved by our platform were 17% and 15.2% respectively, surpassing MaskRCNN (15.2%, 13.9%), YOLOv3 (5.4%, 11.9%), and SOLOv2 (3.1%, 10.4%). Considering detections that exceeded the Intersection over Union threshold of 50%, our platform achieved an impressive F1-score of 44.2% and mAP of 46.3%, outperforming MaskRCNN (41.3%, 43.5%), YOLOv3 (15%, 34.1%), and SOLOv2 (9%, 32.3%). Our platform demonstrated the fastest inference time (88ms), compared to MaskRCNN (90ms), SOLOV2 (100ms), and YOLOv3 (106ms). Finally, the minimal amount of training set demonstrated a good generalisation performance –our architecture successfully identified objects in a frame that were not included in the training or validation frames, indicating its ability to handle out-of-domain scenarios. DISCUSSION: We present our development of an innovative intraoperative SMI to demonstrate the future promise of advanced CV in the surgical domain. Through successful implementation in a microscopic dAVF surgery, our framework demonstrates superior performance over current state-of-the-art segmentation architectures in intraoperative landmark guidance with high sample efficiency, representing the most advanced AI-enabled surgical inference platform to date. Our future goals include transfer learning paradigms for scaling to additional surgery types, addressing clinical and technical limitations for performing real-time decoding, and ultimate enablement of a real-time neurosurgical guidance platform. Frontiers Media S.A. 2023-10-23 /pmc/articles/PMC10626480/ /pubmed/37936949 http://dx.doi.org/10.3389/fsurg.2023.1259756 Text en © 2023 Park, Doiphode, Zhang, Pan, Blue, Shi and Buch. https://creativecommons.org/licenses/by/4.0/This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) (https://creativecommons.org/licenses/by/4.0/) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms. |
spellingShingle | Surgery Park, Jay J. Doiphode, Nehal Zhang, Xiao Pan, Lishuo Blue, Rachel Shi, Jianbo Buch, Vivek P. Developing the surgeon-machine interface: using a novel instance-segmentation framework for intraoperative landmark labelling |
title | Developing the surgeon-machine interface: using a novel instance-segmentation framework for intraoperative landmark labelling |
title_full | Developing the surgeon-machine interface: using a novel instance-segmentation framework for intraoperative landmark labelling |
title_fullStr | Developing the surgeon-machine interface: using a novel instance-segmentation framework for intraoperative landmark labelling |
title_full_unstemmed | Developing the surgeon-machine interface: using a novel instance-segmentation framework for intraoperative landmark labelling |
title_short | Developing the surgeon-machine interface: using a novel instance-segmentation framework for intraoperative landmark labelling |
title_sort | developing the surgeon-machine interface: using a novel instance-segmentation framework for intraoperative landmark labelling |
topic | Surgery |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10626480/ https://www.ncbi.nlm.nih.gov/pubmed/37936949 http://dx.doi.org/10.3389/fsurg.2023.1259756 |
work_keys_str_mv | AT parkjayj developingthesurgeonmachineinterfaceusinganovelinstancesegmentationframeworkforintraoperativelandmarklabelling AT doiphodenehal developingthesurgeonmachineinterfaceusinganovelinstancesegmentationframeworkforintraoperativelandmarklabelling AT zhangxiao developingthesurgeonmachineinterfaceusinganovelinstancesegmentationframeworkforintraoperativelandmarklabelling AT panlishuo developingthesurgeonmachineinterfaceusinganovelinstancesegmentationframeworkforintraoperativelandmarklabelling AT bluerachel developingthesurgeonmachineinterfaceusinganovelinstancesegmentationframeworkforintraoperativelandmarklabelling AT shijianbo developingthesurgeonmachineinterfaceusinganovelinstancesegmentationframeworkforintraoperativelandmarklabelling AT buchvivekp developingthesurgeonmachineinterfaceusinganovelinstancesegmentationframeworkforintraoperativelandmarklabelling |