A novel static and dynamic hand gesture recognition using self organizing map with deep convolutional neural network

Abstract

Gesture recognition has gained a lot of popularity as it allows humans to communicate with real or virtual systems through gestures, offering new and natural interaction modalities. Recent technologies, such as augmented reality (AR) and the Internet of Things (IoT), have witnessed enormous growth in computer applications that focus on human–computer interaction (HCI). However, a few of these tactics make use of a combination of methods, such as image segmentation, pre-processing, and classification. The hessian-based multiscale filtering and YCbCr colour space are used to separate the gesture region to be recognized. A modified marker-controlled watershed method is employed to segment the gesture contour along with the eight-connector graph to increase recognition precision. The proposed hand gesture recognition methodology uses Self Organizing Map (SOM) with Deep Convolutional Neural Network (DCNN) provides better results with fast convergence speed. Experiments were carried out on a dataset of 30 static and 6 dynamic gestures and also evaluated on a publicly available IIITA-ROBITA ISL Gesture Database to show the effectiveness. The results show that the suggested method can recognize gesture classes with 95.63% accuracy rate without significantly affecting the recognition time. The proposed algorithm was then implemented to control household appliances.

Keywords

• hand gesture recognition
• human–computer interaction
• hessian-based multiscale filtering
• modified marker-controlled watershed algorithm
• eight-connected filling algorithm
• SOM-based deep convolutional neural network