Responsive-ExtendedHand: Adaptive Visuo-Haptic Feedback Recognizing Object Property With RGB-D Camera for Projected Extended Hand

Abstract

The ExtendedHand interface displays computer graphics (CG) hand images in real space from a projector, allowing the user to visually point at and touch real objects that are out of their physical reach. Furthermore, when the projected CG hand (extended hand) touches an object, the user can feel the tactile sensation of the object through pseudo-haptics by giving the extended hand visual effects that emphasize the action. In the previous psychological study, the human operator had to manually assign the location and shape of objects and the intensities of their visual effects in advance in order to emphasize the appropriate visual effect for the object touched by the extended hand. To increase practical feasibility, we propose an adaptive system that utilizes an RGB-D camera and deep neural networks to generate the appropriate visual effects automatically and apply them to the projected extended hand. By employing U-Net to generate the appropriate intensities of the visual effects from the captured color and depth images, the system can estimate the appropriate visual effects for objects without pre-setting them. The user evaluation results showed that the proposed system allowed users to naturally perceive the tactile sensation of objects at a rate of 44%, instead of the manual rate of 49%.

Keywords

• Augmented reality
• body augmentation
• deep neural network
• pseudo-haptics