How does hand gestures in videos impact social media engagement - Insights based on deep learning

Abstract

With the fast improvement and development in deep learning and computer vision, the interaction between humans and computers is becoming immense. This research work aims at recognizing hand gestures performed in TEDx videos and analyse the relationship between gestures and user engagement. Here we propose a technique based on deep learning, Convolutional Neural Network (CNN), to recognize hand gestures from a video or image input. ResNeXt-101 model is used for the classification of hand gestures. Here we also used images from the Twenty Bn dataset for some gestures while we also collected gesture images from TEDx videos. Our experiments obtained high accuracy for gesture recognition on training as 95% to 99% and 94.35% for testing. Firstly a gesture is identified in each frame and then classified. We also created a count function that helped us count the number of times a gesture was performed and help us analyse these talks in a better way. Two experimental studies were carried out to analyse viewer engagement: one, the effect of suitable gestures on the viewer count, and secondly, suitable gestures on the sentiment of the viewer. Interesting results are observed that suitable gestures from talkers have an impact on increasing positive review and viewer count.

Keywords

• Computer vision
• Hand gesture
• Deep learning
• Hand gesture recognition
• Neural networks
• Convolutional neural network