Cargando…
Biomimetic Incremental Domain Generalization with a Graph Network for Surgical Scene Understanding
Surgical scene understanding is a key barrier for situation-aware robotic surgeries and the associated surgical training. With the presence of domain shifts and the inclusion of new instruments and tissues, learning domain generalization (DG) plays a pivotal role in expanding instrument–tissue inter...
Autores principales: | , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
MDPI
2022
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9220121/ https://www.ncbi.nlm.nih.gov/pubmed/35735584 http://dx.doi.org/10.3390/biomimetics7020068 |
_version_ | 1784732294328614912 |
---|---|
author | Seenivasan, Lalithkumar Islam, Mobarakol Ng, Chi-Fai Lim, Chwee Ming Ren, Hongliang |
author_facet | Seenivasan, Lalithkumar Islam, Mobarakol Ng, Chi-Fai Lim, Chwee Ming Ren, Hongliang |
author_sort | Seenivasan, Lalithkumar |
collection | PubMed |
description | Surgical scene understanding is a key barrier for situation-aware robotic surgeries and the associated surgical training. With the presence of domain shifts and the inclusion of new instruments and tissues, learning domain generalization (DG) plays a pivotal role in expanding instrument–tissue interaction detection to new domains in robotic surgery. Mimicking the ability of humans to incrementally learn new skills without forgetting their old skills in a similar domain, we employ incremental DG on scene graphs to predict instrument–tissue interaction during robot-assisted surgery. To achieve incremental DG, incorporate incremental learning (IL) to accommodate new instruments and knowledge-distillation-based student–teacher learning to tackle domain shifts in the new domain. Additionally, we designed an enhanced curriculum by smoothing (E-CBS) based on Laplacian of Gaussian (LoG) and Gaussian kernels, and integrated it with the feature extraction network (FEN) and graph network to improve the instrument–tissue interaction performance. Furthermore, the FEN’s and graph network’s logits are normalized by temperature normalization (T-Norm), and its effect in model calibration was studied. Quantitative and qualitative analysis proved that our incrementally-domain generalized interaction detection model was able to adapt to the target domain (transoral robotic surgery) while retaining its performance in the source domain (nephrectomy surgery). Additionally, the graph model enhanced by E-CBS and T-Norm outperformed other state-of-the-art models, and the incremental DG technique performed better than the naive domain adaption and DG technique. |
format | Online Article Text |
id | pubmed-9220121 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2022 |
publisher | MDPI |
record_format | MEDLINE/PubMed |
spelling | pubmed-92201212022-06-24 Biomimetic Incremental Domain Generalization with a Graph Network for Surgical Scene Understanding Seenivasan, Lalithkumar Islam, Mobarakol Ng, Chi-Fai Lim, Chwee Ming Ren, Hongliang Biomimetics (Basel) Article Surgical scene understanding is a key barrier for situation-aware robotic surgeries and the associated surgical training. With the presence of domain shifts and the inclusion of new instruments and tissues, learning domain generalization (DG) plays a pivotal role in expanding instrument–tissue interaction detection to new domains in robotic surgery. Mimicking the ability of humans to incrementally learn new skills without forgetting their old skills in a similar domain, we employ incremental DG on scene graphs to predict instrument–tissue interaction during robot-assisted surgery. To achieve incremental DG, incorporate incremental learning (IL) to accommodate new instruments and knowledge-distillation-based student–teacher learning to tackle domain shifts in the new domain. Additionally, we designed an enhanced curriculum by smoothing (E-CBS) based on Laplacian of Gaussian (LoG) and Gaussian kernels, and integrated it with the feature extraction network (FEN) and graph network to improve the instrument–tissue interaction performance. Furthermore, the FEN’s and graph network’s logits are normalized by temperature normalization (T-Norm), and its effect in model calibration was studied. Quantitative and qualitative analysis proved that our incrementally-domain generalized interaction detection model was able to adapt to the target domain (transoral robotic surgery) while retaining its performance in the source domain (nephrectomy surgery). Additionally, the graph model enhanced by E-CBS and T-Norm outperformed other state-of-the-art models, and the incremental DG technique performed better than the naive domain adaption and DG technique. MDPI 2022-05-28 /pmc/articles/PMC9220121/ /pubmed/35735584 http://dx.doi.org/10.3390/biomimetics7020068 Text en © 2022 by the authors. https://creativecommons.org/licenses/by/4.0/Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/). |
spellingShingle | Article Seenivasan, Lalithkumar Islam, Mobarakol Ng, Chi-Fai Lim, Chwee Ming Ren, Hongliang Biomimetic Incremental Domain Generalization with a Graph Network for Surgical Scene Understanding |
title | Biomimetic Incremental Domain Generalization with a Graph Network for Surgical Scene Understanding |
title_full | Biomimetic Incremental Domain Generalization with a Graph Network for Surgical Scene Understanding |
title_fullStr | Biomimetic Incremental Domain Generalization with a Graph Network for Surgical Scene Understanding |
title_full_unstemmed | Biomimetic Incremental Domain Generalization with a Graph Network for Surgical Scene Understanding |
title_short | Biomimetic Incremental Domain Generalization with a Graph Network for Surgical Scene Understanding |
title_sort | biomimetic incremental domain generalization with a graph network for surgical scene understanding |
topic | Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9220121/ https://www.ncbi.nlm.nih.gov/pubmed/35735584 http://dx.doi.org/10.3390/biomimetics7020068 |
work_keys_str_mv | AT seenivasanlalithkumar biomimeticincrementaldomaingeneralizationwithagraphnetworkforsurgicalsceneunderstanding AT islammobarakol biomimeticincrementaldomaingeneralizationwithagraphnetworkforsurgicalsceneunderstanding AT ngchifai biomimeticincrementaldomaingeneralizationwithagraphnetworkforsurgicalsceneunderstanding AT limchweeming biomimeticincrementaldomaingeneralizationwithagraphnetworkforsurgicalsceneunderstanding AT renhongliang biomimeticincrementaldomaingeneralizationwithagraphnetworkforsurgicalsceneunderstanding |