A Sample Aggregation Approach to Experiences Replay of Dyna-Q Learning

Abstract

In a complex environment, the learning efficiency of reinforcement learning methods always decreases due to large-scale or continuous spaces problems, which can cause the well-known curse of dimensionality. To deal with this problem and enhance learning efficiency, this paper introduces an aggregation method by using framework of sample aggregation based on Chinese restaurant process (CRP), named FSA-CRP, to cluster experiential samples, which is represented by quadruples of the current state, action, next state, and the obtained reward. In addition, the proposed algorithm applies a similarity estimation method, the MinHash method, to calculate the similarity between samples. Moreover, to improve the learning efficiency, the experience sharing Dyna learning algorithm based on samples/clusters prediction method is proposed. While an agent learns the value function of the current state, it acquires clustering results, the value functions of the sample merge with the original as the updated value function of the cluster. In indirect learning (planning) for the Dyna-Q, a learning agent looks for the most likely branches of the constructed FSA-CRP model to raise up learning efficiency. The most likely branches will be selected by an improved action/sample selection algorithm. The algorithm applies the probability that the sample appears in the cluster to select simulated experiences for indirect learning. To verify the validity and applicability of the proposed method, experiments are conducted on a simulated maze and a cart-pole system. The results demonstrate that the proposed method can effectively accelerate the learning process.

Keywords

• Dyna-Q
• Minhash
• Chinese restaurant process
• FSA-CRP model
• prediction