Orthogonal Adversarial Deep Reinforcement Learning for Discrete- and Continuous-Action Problems

Abstract

Recent advancements in deep neural network (DNN) technology are enhancing the utilities of machine learning and meeting potential demands for real-world applications. Deep reinforcement learning (DRL) is a key component in realizing complex decision-making and control by machine learning. However, training DRL-based policies is challenging due to the high costs and risks associated with interactions with real environments. Frameworks to avoid adversarial vulnerability and to achieve robustness are promising solutions to this problem, as they consider worst-case inputs during the evaluation and training phases. In the field of adversarial DRL, regularization methods are known to be effective in maintaining output consistency even when suffering from adversarial perturbations. However, these methods often introduce an undesired bias that conflicts with the original DRL objective, that is, to maximize accumulated rewards. In this study, we propose a new technique that mitigates the undesirable bias effect by orthogonalizing the regularization term with the DRL objective. This approach allows for the use of relatively large adversarial perturbations during training, resulting in more robust policies without significant computational overhead. We evaluate our technique on benchmarks for both discrete and continuous action domains, including Atari 2600 and PyBullet, by integrating it into popular DRL methods such as deep Q-network (DQN), deep deterministic policy gradient (DDPG), and soft actor-critic (SAC). The results demonstrate that our proposed technique is either superior to or competitive with baseline and state-of-the-art methods across all settings, particularly improving robustness by at least twice scores in DQN and DDPG under the adversarial perturbations.

Keywords

• Deep reinforcement learning
• deep learning
• adversarial machine learning
• robust control