Finding Your Way from the Bed to the Kitchen: Re-enacting and Re-combining Sensorimotor Episodes Learned from Human Demonstration

Frontiers in Robotics and AI. 2016;3 DOI 10.3389/frobt.2016.00009


Journal Homepage

Journal Title: Frontiers in Robotics and AI

ISSN: 2296-9144 (Online)

Publisher: Frontiers Media S.A.

LCC Subject Category: Technology: Mechanical engineering and machinery | Science: Mathematics: Instruments and machines: Electronic computers. Computer science

Country of publisher: Switzerland

Language of fulltext: English

Full-text formats available: PDF, HTML, XML



Erik A. Billing (University of Skövde)
Henrik eSvensson (University of Skövde)
Robert eLowe (University of Skövde)
Robert eLowe (University of Gothenburg)
Tom eZiemke (University of Skövde)
Tom eZiemke (Linköping University)


Blind peer review

Editorial Board

Instructions for authors

Time From Submission to Publication: 14 weeks


Abstract | Full Text

Several simulation theories have been proposed as an explanation for how humans and other agents internalize an inner world that allows them to simulate interactions with the external real world -- prospectively and retrospectively. Such internal simulation of interaction with the environment has been argued to be a key mechanism behind mentalizing and planning. In the present work, we study internal simulations in a robot acting in a simulated human environment. A model of sensory-motor interactions with the environment is generated from human demonstrations, and tested on a Robosoft Kompai robot. The model is used as a controller for the robot, reproducing the demonstrated behavior. Information from several different demonstrations is mixed, allowing the robot to produce novel paths through the environment, towards a goal specified by top-down contextual information. The robot model is also used in a covert mode, where actions are inhibited and perceptions are generated by a forward model. As a result, the robot generates an internal simulation of the sensory-motor interactions with the environment. Similar to the overt mode, the model is able to reproduce the demonstrated behavior as internal simulations. When experiences from several demonstrations are combined with a top-down goal signal, the system produces internal simulations of novel paths through the environment. These results can be understood as the robot imagining an inner world generated from previous experience, allowing it to try out different possible futures without executing actions overtly. We found that the success rate in terms of reaching the specified goal was higher during internal simulation, compared to overt action. These results are linked to a reduction in prediction errors generated during covert action. Despite the fact that the model is quite successful in terms of generating covert behavior towards specified goals, internal simulations display different temporal distributions compared to their overt counterparts. Links to human cognition and specifically mental imagery are discussed.