Efficient Sim-to-Real Transfer in Reinforcement Learning Through Domain Randomization and Domain Adaptation

Abstract

Reinforcement learning has gained significant interest in modern industries for its advancements in tackling challenging control tasks compared to rule-based programs. However, the robustness aspect of this technique is still under development, limiting its widespread adoption. This problem has become more pronounced as users switch to training simulations to reduce costs, resulting in a reality gap that negatively affects real-world performance. One popular method employed to mitigate this problem is randomizing uncertain parameters of the environment during training. Nevertheless, this approach requires expert knowledge to determine the appropriate range of randomization. On the other hand, there is a technique that introduces fine-tuning of agents by adapting their policies to new environments. However, it is challenging to adapt policies when the new environment requires shifting them off their initial distribution. These obstacles limit the practical utilization and popularization of both techniques. Our study proposes a hybrid approach that handles these issues by fine-tuning agents trained with domain randomization through additional real-world training. To assess the efficacy of our approach, we conducted experiments involving a rotary inverted pendulum, augmented with an extra weight not represented in the simulation. Additionally, we employed simulated environments including a cart pole, a simple pendulum, a quadruped, and an ant robot scenario. These environments were given in two distinct versions with mismatching parameters to imitate a gap between training and testing conditions. The results demonstrate that adding as few as twenty to fifty additional real-world training episodes can significantly enhance the performance of agents trained with domain randomization. Moreover, including fifty to two hundred additional episodes can elevate it to a level comparable to those fully trained in the real world. Our study concludes that achieving efficient simulation-to-reality transfer is feasible with domain randomization and relatively small amounts of real-world training.

Keywords

• Distributional shift problem
• domain adaptation
• domain randomization
• reality gap
• reinforcement learning
• robustness