Evading Anti-Malware Engines With Deep Reinforcement Learning

Abstract

To reduce the risks of malicious software, malware detection methods using machine learning have received tremendous attention in recent years. Most of the conventional methods are based on supervised learning, which relies on static features with definite labels. However, recent studies have shown the models based on supervised learning are vulnerable to deliberate attacks. This work tends to expose and demonstrate the weakness in these models. A DQEAF framework using reinforcement learning to evade anti-malware engines is presented. DQEAF trains an AI agent through a neural network by constantly interacting with malware samples. Actions are a set of reasonable modifications, which do not damage samples' structure and functions. The agent selects the optimal sequence of actions to modify the malware samples, thus they can bypass the detection engines. The training process depends on the characteristics of the raw binary stream features of samples. The experiments show that the proposed method has a success rate of 75%. The efficacy of the proposed DQEAF has also been evaluated by other families of malicious software, which shows good robustness.

Keywords

• Anti-malware engines evasion
• deep machine learning
• malware detection
• reinforcement learning