Performance evaluations of AI-based obfuscated and encrypted malicious script detection with feature optimization

Abstract

In the digital security environment, the obfuscation and encryption of mali-cious scripts are primary attack methods used to evade detection. Thesescripts—easily spread through websites, emails, and file downloads—can beautomatically executed on users’ systems, posing serious security threats. Toovercome the limitations of signature-based detection methods, this study pro-posed a methodology for real-time detection of obfuscated and encrypted mali-cious scripts using ML/DL models with feature optimization techniques. Theobfuscated script datasets were analyzed to identify the unique characteristics,classified into 16 feature sets, to evaluate the optimal features for the bestdetection accuracy. Although the detection accuracy of these datasets was <20%, when tested with commercial antivirus services, the experimental resultsusing ML and DL models demonstrated that the proposed light gradient boost-ing model (LGBM) could achieve the best detection accuracy and processingspeed. The LGBM outperformed other artificial intelligence models by achiev-ing 97% accuracy and the minimum processing time in the decoded, obfus-cated, and encrypted dataset cases.

Keywords

• deep learning
• encryption
• feature optimization
• machine learning
• malicious script
• obfuscation