Sign Language Recognition Based on CNN-BiLSTM Using RF Signals

Abstract

As the number of individuals with speech and hearing impairments continues to grow, the demand for sign language recognition systems is increasing. Furthermore, most radio frequency-based gesture recognition systems have primarily focused on simple gestures, neglecting more complex actions such as sign language. Therefore, this paper proposes RF-SL, a commercial RFID-based contactless sign language recognition system. This system does not require users to wear any tags and only requires them to perform sign language gestures positioned between an antenna and a fixed multi-tag array to collect sign language signals. Firstly, by removing static reflection noise, we reduce the impact of environmental noise and obtain purer data. Secondly, to accurately delineate the start and end points of sign language action signals, we propose an improved version of the Varri signal segmentation algorithm called Varri+ to effectively optimize the segmentation of sign language actions. Thirdly, we combine the CNN model, known for its robust feature extraction capabilities, with the BiLSTM model, which excels in feature fusion. Extensive testing and evaluation were conducted in two realistic scenarios. The experimental results show that RF-SL achieves an overall average recognition accuracy of 96.8%, with an average recognition accuracy of 96.3% for new users. Additionally, RF-SL achieves an F1 score of 97.5% in an empty room with the weak multipath effects and an F1 score of 93.6% in a classroom with the strong multipath effects. These results demonstrate that our system is dynamic, flexible and robust in sign language recognition.

Keywords

• Sign language recognition
• RF signals
• signal segmentation
• RFID
• CNN
• BiLSTM