3D Hand Gestures Segmentation and Optimized Classification Using Deep Learning

Abstract

Hand gestures recognition system has massive applications which are mainly utilized in robotics and computer vision specially to control Unmanned Aerial Vehicles (UAV). These methods bypass the presence of electronic control to UAVs and provide an ease to the operators. In this paper, we present a method for 3D hand gestures segmentation and classification by combining MASK-RCNN with Grass Hopper Optimization. We created a private 3D and RGB hand gestures dataset using Intel Kinetic and Intel Real sense d435i camera, then proposed a model for RGB hand gestures to estimate the key points using human kinematics, the key points later then utilize to get the best degree of freedom (DoF). The grass hopper optimization besides minimum distance function was applied to achieve the finest deep features from the 3D hand gestures dataset. The ResNet50 network is used as the backbone to calculate the Overlap Coefficient (OC) for segmentation and the ResNet50, ResNet101 networks to calculate the classification for 3D hand gestures. The classification accuracy achieved on the private dataset is 99.05% and 99.29% on public Microsoft Kinect and Leap Motion dataset where the OC are 88.16%. and 88.19% respectively.

Keywords

• 3D hand gestures
• grass-hopper optimization
• segmentation
• depth images