Three‐dimensional posture estimation of robot forceps using endoscope with convolutional neural network

BACKGROUND: In recent years, there has been significant developments in surgical robots. Image‐based sensing of surgical instruments, without the use of electric sensors, are preferred for easily washable robots. METHODS: We propose a method to estimate the three‐dimensional posture of the tip of the forceps tip by using an endoscopic image. A convolutional neural network (CNN) receives the image of the tracked markers attached to the forceps as an input and outputs the posture of the forceps. RESULTS: The posture estimation results showed that the posture estimated from the image followed the electrical sensor. The estimated results of the external force calculated based on the posture also followed the measured values. CONCLUSION: The method which estimates the forceps posture from the image using CNN is effective. The mean absolute error of the estimated external force is smaller than the human detection limit.