Robust Incremental Outlier Detection Approach Based on a New Metric in Data Streams

Abstract

Detecting outliers in real time from multivariate streaming data is a vital and challenging research topic in many areas. Recently introduced the incremental Local Outlier Factor (iLOF) approach and its variants have received considerable attention as they achieve high detection performance in data streams with varying distributions. However, these iLOF-based approaches still have some major limitations: i) Poor detection in high-dimensional data; ii) The difficulty of determining the proper nearest neighbor number $k$ ; iii) Instead of labeling the outlier, assigning a score to each sample that indicates the probability to be an outlier; iv) Inability to detect a long sequence (small cluster) of outliers. This article proposes a new robust outlier detection method (RiLOF) based on iLOF that can effectively overcome these limitations. In the RiLOF method, a novel metric called Median of Nearest Neighborhood Absolute Deviation (MoNNAD) has been developed that uses the median of the local absolute deviation of the samples LOF values. Unlike the previously reported LOF-based approaches, RiLOF is capable of achieving outlier detection in different data stream applications using the same hyperparameters. Extensive experiments performed on 15 different real-world data sets demonstrate that RiLOF remarkably outperforms 12 different state-of-the-art competitors.

Keywords

• Incremental learning
• local outlier factor (LOF)
• new metric
• outlier detection
• robustness