DRL-SRS: A Deep Reinforcement Learning Approach for Optimizing Spaced Repetition Scheduling

Abstract

Optimizing spaced repetition schedules is of great importance for enhancing long-term memory retention in both real-world applications, e.g., online learning platforms, and academic applications, e.g., cognitive science. Traditional methods tackle this problem by employing handcrafted rules while modern methods try to optimize scheduling using deep reinforcement learning (DRL). Existing DRL-based approaches model the problem by selecting the optimal next item to appear, which implies the learner can only learn one item in a day. However, the most essential point to enhancing long-term memory is to select the optimal interval to review. To this end, we present a novel approach to DRL to optimize spaced repetition scheduling. The contribution of our framework is three-fold. We first introduce a Transformer-based model to estimate the recall probability of a learning item accurately, which encodes the temporal dynamics of a learner’s learning trajectories. Second, we build a simulation environment based on our recall probability estimation model. Third, we utilize the Deep Q-Network (DQN) as the agent to learn the optimal review intervals for learning items and train the policy in a recurrent manner. Experimental results demonstrate that our framework achieves state-of-the-art performance against competing methods. Our method achieves an MAE (mean average error) score of 0.0274 on a memory prediction task, which is 11% lower than the second-best method. For spaced repetition scheduling, our method achieves mean recall probabilities of 0.92, 0.942, and 0.372 in three different environments, the best performance in all scenarios.

Keywords

• spaced repetition
• deep reinforcement learning
• memory models
• half-life regression
• transformers