Adaptive Optimization in Evolutionary Reinforcement Learning Using Evolutionary Mutation Rates

Abstract

Deep reinforcement learning (DRL) has achieved notable success in continuous control tasks. However, it faces challenges that limit its applicability to a wider array of tasks, including sparse rewards and limited exploration. Recently, the integration of evolutionary algorithms (EAs) with deep reinforcement learning has emerged as a significant area of research. Evolutionary reinforcement learning (ERL) methods can help address specific challenges inherent in conventional reinforcement learning algorithms. However, the introduction of evolutionary computation algorithms increases the number of hyperparameters, and sensitivity to these hyperparameters continues to pose a significant challenge. This paper proposes an evolutionary reinforcement learning method incorporating evolutionary mutation rates. This method integrates a self-adaptive mutation rate mechanism into the ERL framework, which maintains two populations: one consisting of individuals (agents) and the other one comprising mutation rates. This represents our original contribution to this research. The actor population is categorized into several groups, each assigned a specific mutation rate. After mutation, the mutation rate of the population evolves based on the performance of the mutations within the actor population. This approach addresses the challenge of selecting mutation rates in ERL. Experimental results demonstrate superior performance compared to the standard ERL framework across six continuous control tasks.

Keywords

• Deep reinforcement learning
• evolutionary computation algorithm
• evolutionary reinforcement learning
• self-adaptive mutation
• optimization