An Recognition–Verification Mechanism for Real-Time Chinese Sign Language Recognition Based on Multi-Information Fusion

For online sign language recognition (SLR) based on inertial measurement unit (IMU) and a surface electromyography (sEMG) sensor, achieving high-accuracy is a major challenge. The traditional method for that is the segmentation–recognition mechanism, which has two key challenges: (1) it is difficult to design a highly robust segmentation method for online data with inconspicuous segmentation information; and (2) the diversity of input data will increase the burden of the classification. The recognition–verification mechanism was proposed to improve the performance of online SLR. In the recognition stage, we used sliding windows to pull the data, and applied a convolutional neural network (CNN) to classify the sign language signal. In the verification stage, the confidence was evaluated by the Siamese network to judge the correctness of the classification results. The accuracy and rapidity of the classification model were discussed for 86 categories of Chinese sign language. In the experiments for online SLR, the superiority of the recognition–verification mechanism compared to the segmentation–recognition mechanism was verified.