An End-to-End Stochastic Action and Visual Estimation System Towards Autonomous Teleoperation

Abstract

Teleoperation systems have been getting significant attention from many application areas for decades. However, classical teleoperation systems suffer from problems such as lack of natural feedback, latency, and inefficient operator throughput. Researchers attempted to address these issues by performing some of the teleoperation sub-tasks autonomously whenever requested by the operator. Nevertheless, these systems still need the operator to see the need for autonomous actions and initiate these actions manually, which is demanding for the operators. This paper proposes a novel end-to-end Stochastic Assistive Teleoperation System (SATS) that always stays in the loop, automatically detects applicable actions with probabilities, and produces visual scene estimations for each of these actions, which results in increased operator efficiency and throughput. We introduce several methods that combine ideas from Markov processes and recurrent neural networks to stochastically predict future action sequences and scene configurations with tractable algorithms. Experiments performed with a group of operators on real and simulative teleoperation environments show that operators issue a considerably smaller number of commands compared to alternative methods. We also showed that the operators can manipulate multiple robots simultaneously using our technique, which boosts the operator throughput even further. We provide supplementary video material that demonstrates SATS in action.

Keywords

• Action-conditioned prediction
• assistive teleoperation
• Markov chains
• recurrent neural networks
• semantic video segmentation
• stochastic video prediction