Cargando…

Biomimetic Incremental Domain Generalization with a Graph Network for Surgical Scene Understanding

Surgical scene understanding is a key barrier for situation-aware robotic surgeries and the associated surgical training. With the presence of domain shifts and the inclusion of new instruments and tissues, learning domain generalization (DG) plays a pivotal role in expanding instrument–tissue inter...

Descripción completa

Detalles Bibliográficos
Autores principales: Seenivasan, Lalithkumar, Islam, Mobarakol, Ng, Chi-Fai, Lim, Chwee Ming, Ren, Hongliang
Formato: Online Artículo Texto
Lenguaje:English
Publicado: MDPI 2022
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9220121/
https://www.ncbi.nlm.nih.gov/pubmed/35735584
http://dx.doi.org/10.3390/biomimetics7020068
_version_ 1784732294328614912
author Seenivasan, Lalithkumar
Islam, Mobarakol
Ng, Chi-Fai
Lim, Chwee Ming
Ren, Hongliang
author_facet Seenivasan, Lalithkumar
Islam, Mobarakol
Ng, Chi-Fai
Lim, Chwee Ming
Ren, Hongliang
author_sort Seenivasan, Lalithkumar
collection PubMed
description Surgical scene understanding is a key barrier for situation-aware robotic surgeries and the associated surgical training. With the presence of domain shifts and the inclusion of new instruments and tissues, learning domain generalization (DG) plays a pivotal role in expanding instrument–tissue interaction detection to new domains in robotic surgery. Mimicking the ability of humans to incrementally learn new skills without forgetting their old skills in a similar domain, we employ incremental DG on scene graphs to predict instrument–tissue interaction during robot-assisted surgery. To achieve incremental DG, incorporate incremental learning (IL) to accommodate new instruments and knowledge-distillation-based student–teacher learning to tackle domain shifts in the new domain. Additionally, we designed an enhanced curriculum by smoothing (E-CBS) based on Laplacian of Gaussian (LoG) and Gaussian kernels, and integrated it with the feature extraction network (FEN) and graph network to improve the instrument–tissue interaction performance. Furthermore, the FEN’s and graph network’s logits are normalized by temperature normalization (T-Norm), and its effect in model calibration was studied. Quantitative and qualitative analysis proved that our incrementally-domain generalized interaction detection model was able to adapt to the target domain (transoral robotic surgery) while retaining its performance in the source domain (nephrectomy surgery). Additionally, the graph model enhanced by E-CBS and T-Norm outperformed other state-of-the-art models, and the incremental DG technique performed better than the naive domain adaption and DG technique.
format Online
Article
Text
id pubmed-9220121
institution National Center for Biotechnology Information
language English
publishDate 2022
publisher MDPI
record_format MEDLINE/PubMed
spelling pubmed-92201212022-06-24 Biomimetic Incremental Domain Generalization with a Graph Network for Surgical Scene Understanding Seenivasan, Lalithkumar Islam, Mobarakol Ng, Chi-Fai Lim, Chwee Ming Ren, Hongliang Biomimetics (Basel) Article Surgical scene understanding is a key barrier for situation-aware robotic surgeries and the associated surgical training. With the presence of domain shifts and the inclusion of new instruments and tissues, learning domain generalization (DG) plays a pivotal role in expanding instrument–tissue interaction detection to new domains in robotic surgery. Mimicking the ability of humans to incrementally learn new skills without forgetting their old skills in a similar domain, we employ incremental DG on scene graphs to predict instrument–tissue interaction during robot-assisted surgery. To achieve incremental DG, incorporate incremental learning (IL) to accommodate new instruments and knowledge-distillation-based student–teacher learning to tackle domain shifts in the new domain. Additionally, we designed an enhanced curriculum by smoothing (E-CBS) based on Laplacian of Gaussian (LoG) and Gaussian kernels, and integrated it with the feature extraction network (FEN) and graph network to improve the instrument–tissue interaction performance. Furthermore, the FEN’s and graph network’s logits are normalized by temperature normalization (T-Norm), and its effect in model calibration was studied. Quantitative and qualitative analysis proved that our incrementally-domain generalized interaction detection model was able to adapt to the target domain (transoral robotic surgery) while retaining its performance in the source domain (nephrectomy surgery). Additionally, the graph model enhanced by E-CBS and T-Norm outperformed other state-of-the-art models, and the incremental DG technique performed better than the naive domain adaption and DG technique. MDPI 2022-05-28 /pmc/articles/PMC9220121/ /pubmed/35735584 http://dx.doi.org/10.3390/biomimetics7020068 Text en © 2022 by the authors. https://creativecommons.org/licenses/by/4.0/Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
spellingShingle Article
Seenivasan, Lalithkumar
Islam, Mobarakol
Ng, Chi-Fai
Lim, Chwee Ming
Ren, Hongliang
Biomimetic Incremental Domain Generalization with a Graph Network for Surgical Scene Understanding
title Biomimetic Incremental Domain Generalization with a Graph Network for Surgical Scene Understanding
title_full Biomimetic Incremental Domain Generalization with a Graph Network for Surgical Scene Understanding
title_fullStr Biomimetic Incremental Domain Generalization with a Graph Network for Surgical Scene Understanding
title_full_unstemmed Biomimetic Incremental Domain Generalization with a Graph Network for Surgical Scene Understanding
title_short Biomimetic Incremental Domain Generalization with a Graph Network for Surgical Scene Understanding
title_sort biomimetic incremental domain generalization with a graph network for surgical scene understanding
topic Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9220121/
https://www.ncbi.nlm.nih.gov/pubmed/35735584
http://dx.doi.org/10.3390/biomimetics7020068
work_keys_str_mv AT seenivasanlalithkumar biomimeticincrementaldomaingeneralizationwithagraphnetworkforsurgicalsceneunderstanding
AT islammobarakol biomimeticincrementaldomaingeneralizationwithagraphnetworkforsurgicalsceneunderstanding
AT ngchifai biomimeticincrementaldomaingeneralizationwithagraphnetworkforsurgicalsceneunderstanding
AT limchweeming biomimeticincrementaldomaingeneralizationwithagraphnetworkforsurgicalsceneunderstanding
AT renhongliang biomimeticincrementaldomaingeneralizationwithagraphnetworkforsurgicalsceneunderstanding