Cargando…

Developing the surgeon-machine interface: using a novel instance-segmentation framework for intraoperative landmark labelling

INTRODUCTION: The utilisation of artificial intelligence (AI) augments intraoperative safety, surgical training, and patient outcomes. We introduce the term Surgeon-Machine Interface (SMI) to describe this innovative intersection between surgeons and machine inference. A custom deep computer vision...

Descripción completa

Detalles Bibliográficos
Autores principales: Park, Jay J., Doiphode, Nehal, Zhang, Xiao, Pan, Lishuo, Blue, Rachel, Shi, Jianbo, Buch, Vivek P.
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Frontiers Media S.A. 2023
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10626480/
https://www.ncbi.nlm.nih.gov/pubmed/37936949
http://dx.doi.org/10.3389/fsurg.2023.1259756
_version_ 1785131345618403328
author Park, Jay J.
Doiphode, Nehal
Zhang, Xiao
Pan, Lishuo
Blue, Rachel
Shi, Jianbo
Buch, Vivek P.
author_facet Park, Jay J.
Doiphode, Nehal
Zhang, Xiao
Pan, Lishuo
Blue, Rachel
Shi, Jianbo
Buch, Vivek P.
author_sort Park, Jay J.
collection PubMed
description INTRODUCTION: The utilisation of artificial intelligence (AI) augments intraoperative safety, surgical training, and patient outcomes. We introduce the term Surgeon-Machine Interface (SMI) to describe this innovative intersection between surgeons and machine inference. A custom deep computer vision (CV) architecture within a sparse labelling paradigm was developed, specifically tailored to conceptualise the SMI. This platform demonstrates the ability to perform instance segmentation on anatomical landmarks and tools from a single open spinal dural arteriovenous fistula (dAVF) surgery video dataset. METHODS: Our custom deep convolutional neural network was based on SOLOv2 architecture for precise, instance-level segmentation of surgical video data. Test video consisted of 8520 frames, with sparse labelling of only 133 frames annotated for training. Accuracy and inference time, assessed using F1-score and mean Average Precision (mAP), were compared against current state-of-the-art architectures on a separate test set of 85 additionally annotated frames. RESULTS: Our SMI demonstrated superior accuracy and computing speed compared to these frameworks. The F1-score and mAP achieved by our platform were 17% and 15.2% respectively, surpassing MaskRCNN (15.2%, 13.9%), YOLOv3 (5.4%, 11.9%), and SOLOv2 (3.1%, 10.4%). Considering detections that exceeded the Intersection over Union threshold of 50%, our platform achieved an impressive F1-score of 44.2% and mAP of 46.3%, outperforming MaskRCNN (41.3%, 43.5%), YOLOv3 (15%, 34.1%), and SOLOv2 (9%, 32.3%). Our platform demonstrated the fastest inference time (88ms), compared to MaskRCNN (90ms), SOLOV2 (100ms), and YOLOv3 (106ms). Finally, the minimal amount of training set demonstrated a good generalisation performance –our architecture successfully identified objects in a frame that were not included in the training or validation frames, indicating its ability to handle out-of-domain scenarios. DISCUSSION: We present our development of an innovative intraoperative SMI to demonstrate the future promise of advanced CV in the surgical domain. Through successful implementation in a microscopic dAVF surgery, our framework demonstrates superior performance over current state-of-the-art segmentation architectures in intraoperative landmark guidance with high sample efficiency, representing the most advanced AI-enabled surgical inference platform to date. Our future goals include transfer learning paradigms for scaling to additional surgery types, addressing clinical and technical limitations for performing real-time decoding, and ultimate enablement of a real-time neurosurgical guidance platform.
format Online
Article
Text
id pubmed-10626480
institution National Center for Biotechnology Information
language English
publishDate 2023
publisher Frontiers Media S.A.
record_format MEDLINE/PubMed
spelling pubmed-106264802023-11-07 Developing the surgeon-machine interface: using a novel instance-segmentation framework for intraoperative landmark labelling Park, Jay J. Doiphode, Nehal Zhang, Xiao Pan, Lishuo Blue, Rachel Shi, Jianbo Buch, Vivek P. Front Surg Surgery INTRODUCTION: The utilisation of artificial intelligence (AI) augments intraoperative safety, surgical training, and patient outcomes. We introduce the term Surgeon-Machine Interface (SMI) to describe this innovative intersection between surgeons and machine inference. A custom deep computer vision (CV) architecture within a sparse labelling paradigm was developed, specifically tailored to conceptualise the SMI. This platform demonstrates the ability to perform instance segmentation on anatomical landmarks and tools from a single open spinal dural arteriovenous fistula (dAVF) surgery video dataset. METHODS: Our custom deep convolutional neural network was based on SOLOv2 architecture for precise, instance-level segmentation of surgical video data. Test video consisted of 8520 frames, with sparse labelling of only 133 frames annotated for training. Accuracy and inference time, assessed using F1-score and mean Average Precision (mAP), were compared against current state-of-the-art architectures on a separate test set of 85 additionally annotated frames. RESULTS: Our SMI demonstrated superior accuracy and computing speed compared to these frameworks. The F1-score and mAP achieved by our platform were 17% and 15.2% respectively, surpassing MaskRCNN (15.2%, 13.9%), YOLOv3 (5.4%, 11.9%), and SOLOv2 (3.1%, 10.4%). Considering detections that exceeded the Intersection over Union threshold of 50%, our platform achieved an impressive F1-score of 44.2% and mAP of 46.3%, outperforming MaskRCNN (41.3%, 43.5%), YOLOv3 (15%, 34.1%), and SOLOv2 (9%, 32.3%). Our platform demonstrated the fastest inference time (88ms), compared to MaskRCNN (90ms), SOLOV2 (100ms), and YOLOv3 (106ms). Finally, the minimal amount of training set demonstrated a good generalisation performance –our architecture successfully identified objects in a frame that were not included in the training or validation frames, indicating its ability to handle out-of-domain scenarios. DISCUSSION: We present our development of an innovative intraoperative SMI to demonstrate the future promise of advanced CV in the surgical domain. Through successful implementation in a microscopic dAVF surgery, our framework demonstrates superior performance over current state-of-the-art segmentation architectures in intraoperative landmark guidance with high sample efficiency, representing the most advanced AI-enabled surgical inference platform to date. Our future goals include transfer learning paradigms for scaling to additional surgery types, addressing clinical and technical limitations for performing real-time decoding, and ultimate enablement of a real-time neurosurgical guidance platform. Frontiers Media S.A. 2023-10-23 /pmc/articles/PMC10626480/ /pubmed/37936949 http://dx.doi.org/10.3389/fsurg.2023.1259756 Text en © 2023 Park, Doiphode, Zhang, Pan, Blue, Shi and Buch. https://creativecommons.org/licenses/by/4.0/This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) (https://creativecommons.org/licenses/by/4.0/) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
spellingShingle Surgery
Park, Jay J.
Doiphode, Nehal
Zhang, Xiao
Pan, Lishuo
Blue, Rachel
Shi, Jianbo
Buch, Vivek P.
Developing the surgeon-machine interface: using a novel instance-segmentation framework for intraoperative landmark labelling
title Developing the surgeon-machine interface: using a novel instance-segmentation framework for intraoperative landmark labelling
title_full Developing the surgeon-machine interface: using a novel instance-segmentation framework for intraoperative landmark labelling
title_fullStr Developing the surgeon-machine interface: using a novel instance-segmentation framework for intraoperative landmark labelling
title_full_unstemmed Developing the surgeon-machine interface: using a novel instance-segmentation framework for intraoperative landmark labelling
title_short Developing the surgeon-machine interface: using a novel instance-segmentation framework for intraoperative landmark labelling
title_sort developing the surgeon-machine interface: using a novel instance-segmentation framework for intraoperative landmark labelling
topic Surgery
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10626480/
https://www.ncbi.nlm.nih.gov/pubmed/37936949
http://dx.doi.org/10.3389/fsurg.2023.1259756
work_keys_str_mv AT parkjayj developingthesurgeonmachineinterfaceusinganovelinstancesegmentationframeworkforintraoperativelandmarklabelling
AT doiphodenehal developingthesurgeonmachineinterfaceusinganovelinstancesegmentationframeworkforintraoperativelandmarklabelling
AT zhangxiao developingthesurgeonmachineinterfaceusinganovelinstancesegmentationframeworkforintraoperativelandmarklabelling
AT panlishuo developingthesurgeonmachineinterfaceusinganovelinstancesegmentationframeworkforintraoperativelandmarklabelling
AT bluerachel developingthesurgeonmachineinterfaceusinganovelinstancesegmentationframeworkforintraoperativelandmarklabelling
AT shijianbo developingthesurgeonmachineinterfaceusinganovelinstancesegmentationframeworkforintraoperativelandmarklabelling
AT buchvivekp developingthesurgeonmachineinterfaceusinganovelinstancesegmentationframeworkforintraoperativelandmarklabelling