Research on Continuous Dynamic Gesture Recognition of Chinese Sign Language Based on Multi-Mode Fusion

Abstract

To solve the problem of the low recognition rate of continuous dynamic gestures in Chinese sign language, a non-invasive end-to-end continuous dynamic gesture recognition system combining Inertial Measurement Unit (IMU) signal and surface electromyography (sEMG) signal is constructed. After the preprocessing and fusion of the IMU signal and sEMG signal, the fusion gesture features which include acceleration, angular velocity, attitude quaternion, and sEMG information are extracted. A network model based on the Bi-directional Long-Short Term Memory (BiLSTM) network and Connectionist Temporal Classification (CTC) as loss function is then constructed, which avoids the adverse effects of inaccurate pre-segmentation of gesture sequences on continuous dynamic gesture recognition and realizes the end-to-end continuous dynamic gesture recognition. Finally, the Chinese Sign Language Database (CSLD), containing 10 kinds of Chinese sign language and 20000 gesture samples, is established. The training and testing of the database indicate that the average recognition rate reaches 98.66%, confirming the superiority of this method.

Keywords

• Gesture recognition
• multimode fusion
• surface electromyography
• inertial measurement unit
• bi-directional long short term memory network