Estimation and control using sampling-based Bayesian reinforcement learning

Abstract

Real-world autonomous systems operate under uncertainty about both their pose and dynamics. Autonomous control systems must simultaneously perform estimation and control tasks to maintain robustness to changing dynamics or modelling errors. However, information gathering actions often conflict with optimal actions for reaching control objectives, requiring a trade-off between exploration and exploitation. The specific problem setting considered here is for discrete-time non-linear systems, with process noise, input-constraints, and parameter uncertainty. This study frames this problem as a Bayes-adaptive Markov decision process and solves it online using Monte Carlo tree search with an unscented Kalman filter to account for process noise and parameter uncertainty. This method is compared with certainty equivalent model predictive control and a tree search method that approximates the QMDP solution, providing insight into when information gathering is useful. Discrete time simulations characterise performance over a range of process noise and bounds on unknown parameters. An offline optimisation method is used to select the Monte Carlo tree search parameters without hand-tuning. In lieu of recursive feasibility guarantees, a probabilistic bounding heuristic is offered that increases the probability of keeping the state within a desired region.

Keywords

• linear systems
• tree searching
• path planning
• belief networks
• bayes methods
• discrete time systems
• decision theory
• stochastic systems
• monte carlo methods
• kalman filters
• markov processes
• mobile robots
• probability
• predictive control
• optimisation
• uncertain systems
• real-world autonomous systems
• pose dynamics
• autonomous control systems
• control tasks
• changing dynamics
• modelling errors
• information gathering actions
• optimal actions
• control objectives
• discrete-time nonlinear systems
• process noise
• input-constraints
• parameter uncertainty
• bayes-adaptive markov decision process
• unscented kalman filter
• certainty equivalent model predictive control
• discrete time simulations
• offline optimisation method
• monte carlo tree search parameters
• sampling-based bayesian reinforcement learning
• recursive feasibility guarantees
• probabilistic bounding heuristic
• qmdp solution