Detection of randomized bot command and control traffic on an end-point host

Abstract

Bots are malicious software entities that unobtrusively infect machines and silently engage in activities ranging from data stealing to cyber warfare. Most recent bot detection methods rely on regularity of bot command and control (C&C) traffic for bot detection but state-of-the-art bots randomize traffic properties to evade regularity based detection techniques. We propose a bot detection system that aims to detect randomized bot C&C traffic and also aim at early bot detection. To this end, separate strategies are devised for bot detection: (i) over a user session and (ii) time periods larger than a user session. Normal HTTP traffic and bot control traffic are modeled over a user session and a Multi-Layer Perceptron Classifier is trained on the two models and later used to classify unlabeled destinations as benign or malicious. For traffic spanning time intervals larger than a user session, temporal persistence, is used to differentiate between traffic to benign and malicious destinations. Testing with multiple datasets yielded good results.

Keywords

• Botnet detection
• Randomized traffic
• Traffic modeling
• Anomaly based