Digital Intentions in the Fingers: I Know What You Are Doing with Your Smartphone

Abstract

Every day, we make thousands of finger movements on the touchscreen of our smartphones. The same movements might be directed at various distal goals. We can type “What is the weather in Rome?” in Google to acquire information from a weather site, or we may type it on WhatsApp to decide whether to visit Rome with a friend. In this study, we show that by watching an agent’s typing hands, an observer can infer whether the agent is typing on the smartphone to obtain information or to share it with others. The probability of answering correctly varies with age and typing style. According to embodied cognition, we propose that the recognition process relies on detecting subtle differences in the agent’s movement, a skill that grows with sensorimotor competence. We expect that this preliminary work will serve as a starting point for further research on sensorimotor representations of digital actions.

Keywords

• action observation
• action recognition
• digital intentions
• digital affordances
• kinematics