Tensor-Based Online Network Anomaly Detection and Diagnosis

Abstract

This paper presents an online anomaly detection system capable of handling operational network traffic of large networks (such as an ISP). We also aim for an effective and practical diagnosis of anomalies diagnosis to produce actionable intelligence that enables automated response. To achieve these objectives, we use the following approaches. (1) We model the status of the network by a stream of tensors where each tensor cell contains a time series. (2) We detect anomalous tensors at discrete time steps using an unsupervised tensor representation learning model. (3) We produce actionable intelligence by diagnosing anomaly detection results and identifying the abnormal time series that are the most likely causes of each anomaly in the tensor. (4) We further analyze the traffic corresponding to each anomalous time series by an innovative method that extracts and isolates the attack traffic. (5) We provide solutions for streaming data anomaly detection challenges such as large volume, high velocity, seasonality, and concept drift. We apply our approach to the complete test set of UGR data to show its practicality and effectiveness. Not only can we detect and isolate most of the labelled attack traffic, but we also identify many organic attack activities in the UGR data. Our results on the complete UGR dataset show high detection and isolation rates for the labelled attacks in the dataset. We also report on additional organic attacks we detected that were originally labelled as background in the dataset. Our analysis shows that the isolated background traffic represents interesting and potentially malicious behavior and can provide invaluable insight for cyber-threat researchers.

Keywords

• Anomaly detection
• anomaly diagnosis
• convolutional neural network
• autoencoder
• NetFlow