On Early Detection of Anomalous Network Flows

Abstract

There are numerous methods of identifying network-based attacks using machine learning, but processing complexity often constrains it to analyses of previously captured traffic to retroactively identify attacks. This paper investigates machine learning for early detection of attacks in progress with minimal preprocessing. We transform raw network data directly into formats suitable for processing with several machine learning and deep learning models including Random Forest and two-dimensional Convolutional Neural Networks. Many of these models demonstrate high accuracy in detecting a mixture of mostly DoS- and botnet-related types of network-based attacks in five open-source traffic datasets containing packet captures from testbed-generated traffic. We compare our results in post-mortem packet trace analysis to prior works which also analyze these datasets. We compare the features, limitations, complexity, and accuracy of our models to those of prior works. When trained and tested on the same datasets, most models performed very well (>95% accuracy) with Random Forest being the best. We also investigated training time required and testing throughput, and the RF consistently outperformed the other five models.

Keywords

• Convolutional neural network
• deep learning
• k-nearest neighbors%22%2C%22default_operator%22%3A%22AND%22%7D%7D%7D"> <italic xmlns:ali="http://www.niso.org/schemas/ali/1.0/" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">k</italic>-nearest neighbors
• machine learning
• multilayer perceptron
• network security