A Novel Online and Non-Parametric Approach for Drift Detection in Big Data

Abstract

A sizable amount of current literature on online drift detection tools thrive on unrealistic parametric strictures such as normality or on non-parametric methods whose power performance is questionable. Using minimal realistic assumptions such as unimodality, we have strived to proffer an alternative, through a novel application of Bernstein's inequality. Simulations from such parametric densities as Beta and Logitnormal as well as real-data analyses demonstrate this new method's superiority over similar techniques relying on bounds, such as Hoeffding's. Improvements are apparent in terms of higher power, efficient sample sizes, and sensitivity to parameter values.

Keywords

• Change point detection
• non-parametric methods
• Hoeffding’s inequality
• Bernstein’s inequality
• big data
• anomaly detection