Tackling Evolving Botnet Threats: A Gradual Self-Training Neural Network Approach

Abstract

Botnets pose a significant challenge to network security but are difficult to detect because of their dynamic and evolving nature, which limits the effectiveness of conventional supervised neural network detection methods. To address this problem, the present study proposes a novel neural network-based self-training framework for botnet detection, in which pseudo-labels are generated from unlabeled data by a trained classifier, which is iteratively refined over time using a combined dataset containing both training and pseudo-labeled data. Although not all of the generated pseudo-labels are applicable to every botnet, the self-training framework can label unseen botnets with behaviors similar to those of known botnets with high confidence. Several strategies are proposed for enhancing the robustness of the classification performance by minimizing the number of incorrect pseudo-labels, mitigating the effects of erroneous pseudo-labels on the overall performance of the network, and optimizing the proportion of unlabeled data for labeling. Experiments conducted on both synthetic datasets confirm the superiority of the proposed method over the base model, particularly when the training data constitutes only a small portion of the total amount dataset. Subsequent experiments also demonstrate the efficacy of the framework in successfully detecting unseen botnet variants and its commendable performance in real-world campus network traffic.

Keywords

• Botnet detection
• NetFlow
• network security