Enhancing sign language recognition using CNN and SIFT: A case study on Pakistan sign language

Abstract

Millions of people who have trouble hearing rely heavily on sign language as their primary means of communication. As a form of visual language, sign language is primarily used as features by the deaf and hard of hearing to communicate with one another. These features make it challenging for the average person to grasp what is being said. As a result, it becomes more difficult for deaf people to interact with the hearing community. In order to solve this communication barrier, scientists have been working on sign language recognition systems. The purpose of this research is to propose a hand gesture recognition framework for Pakistan Sign Language (PSL) by training a Convolutional Neural Network (CNN) on PSL gesture images obtained with the aid of a testbed built in the laboratory using the Kinect motion sensor for Urdu alphabets. Kinect images of hands were taken in this study under a variety of lighting conditions. A feature vector set was created from hand margin, size, shape, palm center points, and finger position using Scale Invariant Feature Transform (SIFT). Important features of PSL-based sign images were extracted using SIFT and then converted to vector points. The proposed improved CNN model was able to achieve an impressive accuracy of 98.74% on the PSL dataset, demonstrating an impressively low error rate of only 1.26%. A case study is demonstrated in the assessment of the system that has been provided. Finally, a comparison is performed between others and our work to show the efficiency of the proposed framework and legitimize the concept.

Keywords

• Pakistan sign language
• Urdu Alphabets
• Hand gesture recognition
• CNN
• SIFT