Reinforcement Learning-Based Cache Replacement Policies for Multicore Processors

Abstract

High-performance computing (HPC) systems need to handle ever-increasing data sizes for fast processing and quick response times. However, modern processors’ caches are unable to handle massive amounts of data, leading to significant cache miss penalties that affect performance. In this context, selecting an effective cache replacement policy is crucial to improving HPC performance. Existing cache replacement policies fall short of Bélády’s optimal algorithm, and we propose a new approach that leverages the coherence state and sharers’ bit-vector of a cache block to make better decisions. We suggest a reinforcement learning-based strategy that learns from past eviction decisions and applies this knowledge to make better decisions in the future. Our approach uses a next-attempt method that combines the results from classic cache replacement algorithms with reinforcement learning. We evaluated our approach using the Sniper simulator and seven kernels from CAP Benchmarks. Our results show that our approach can significantly reduce the cache miss rate by 41.20% and 27.30% in L1 and L2 caches, respectively. In addition, our approach can improve the IPC by 27.33% in the best case and reduce energy consumption by 20.36% compared to an unmodified policy.

Keywords

• Cache replacement
• coherence
• multicore
• reinforcement learning