Dynamic Spectrum Access with Deep Q-learning in Densely Occupied and Partially Observable Environments

Abstract

In this paper, we propose a new Dynamic Spectrum Access (DSA) method for multi-channel wireless networks. We assume that DSA nodes do not have prior knowledge of the system dynamics, and have only partial observability of the channels. Thus, the problem is formulated as a Partially Observable Markov Decision Process (POMDP) with exponential time complexity. We have developed a novel Deep Reinforcement Learning (DRL) based DSA method which combines a double deep Q-learning architecture with a recurrent neural network and takes advantage of a prioritized experience buffer. The simulation analysis shows that the proposed method accurately predicts a channel state based on the fixed-length history of partial observations. Compared with other DRL methods for DSA, the proposed solution can find a near-optimal policy in a smaller number of iterations and suits a wider range of communication environments, including dynamic ones, where channel occupancy pattern changes over time. The performance improvement increases with the number of channels and with a channel state transition uncertainty. To boost the performance of the algorithm in densely occupied environments, multiple DRL exploration strategies are examined and evaluation results are presented in the paper.

Keywords

• cognitive radio
• reinforcement learning
• q-learning