Massively High-Throughput Reinforcement Learning for Classic Control on GPUs

Abstract

This study presents a novel massively high-throughput reinforcement learning (RL) framework specifically designed for addressing classic control problems, leveraging our proposed architecture and algorithms optimized for efficient concurrent computations on GPUs. Our research demonstrates the effectiveness of our methods in efficiently training RL agents across various classic control problems, encompassing both discrete and continuous domains, while achieving rapid and stable performance up to 10K concurrent environment instances. Furthermore, we observe that RL exploration with a large number of parallel instances significantly enhances the stability of updating a shared model. For instance, we show that the stability of Deep Deterministic Policy Gradient (DDPG) training can be achieved without requiring experience replay, as evidenced in our study.

Keywords

• Classic control
• GPU acceleration
• high-throughput
• reinforcement learning