Backhand-View-Based Continuous-Signed-Letter Recognition Using a Rewound Video Sequence and the Previous Signed-Letter Information

Abstract

In sign language, when signed letters are continuously spelled based on backhand view, a previous signed letter influences the trajectory of hand and fingers approaching the pause duration for signing the current signed letter. Since those varied trajectories are regarded as parts of the current signed letter, hand gesture during pause duration of the current signed letter is regarded as insufficient for recognition of the current signed letter. The previous signed letters, and trajectories of hand and fingers between the previous and the current signed letters should be included as data for classification. This paper proposes a method of backhand-view-based continuous-signed-letter recognition using a rewound video sequence with previous signed letter. In the method, a hand shape of previous signed letter and trajectories of finger joints moving from the previous signed letter to the current one are detected, features are then extracted, and finally, the features are classified for signed letter recognition. To evaluate performance of the proposed method, experiments with 10 participants were performed 20 times each, and the results revealed 96.07% accuracy approximately which were improved significantly from the conventional methods using forehand and backhand.

Keywords

• Continuous signed letter
• rewound video
• previous signed-letter
• backhand view
• LSTM