Evading control flow graph based GNN malware detectors via active opcode insertion method with maliciousness preserving

Abstract

Abstract With the continuous advancement of machine learning, numerous malware detection methods that leverage this technology have emerged, presenting new challenges to the generation of adversarial malware. Existing function-preserving adversarial attacks fall short of effectively modifying portable executable (PE) malware control flow graphs (CFGs), thereby failing to bypass the graph neural network (GNN) models that utilize CFGs for detection. To solve this issue, we introduce a novel base modification method called active opcode insertion, which modifies PE CFGs while preserving functionality by inserting a processed sequence of benign and jump opcodes to connect with the original base block. Using reinforcement learning, MalAOI identifies optimal insertion points and benign opcode sequences to autonomously generate adversarial malware that evades GNN model detection. We tested our approach on the BODMAS and SOREL-20M datasets, and the results demonstrate that MalAOI-generated adversarial malware achieves an average evasion rate of 93.73% against the GNN detection model, with only 12.87% increase in byte size.

Keywords

• Malware Detection
• Reinforcement Learning
• Graph Neural Networks
• Control Flow Graphs
• Function-preserving