Featureless Discovery of Correlated and False Intrusion Alerts

Abstract

Malware and cyber-attacks cause substantial damage to corporations. A common countermeasure is Intrusion Detection Systems (IDSs). Unfortunately, IDSs typically raise many alerts on a single incident, with redundant information, and false alerts that are only noise to analysts. For out-of-the-box performance, the impact is so large that alerts are of limited practical use. Existing solutions rely heavily on domain expertise, in feature engineering procedures and explicit algorithms. This has substantial negative impact on the costs of development, deployment, and maintenance. Using feature engineering as part of a method boosts classification metrics, but requires substantial investment, of data science and security expertise, for each deployment. We find that reliance on domain expertise and feature engineering severely inhibits the feasibility of applying existing correlation and filtering methods in practice. To address this, we propose a novel approach for correlating and filtering, with the constraints that methods must be without feature engineering and methods must consume alerts as text strings. Two implementations are presented and evaluated on a partly private and on a public data set. Our implementations are unable to compete with existing methods on common detection metrics, suggesting that investing feature engineering pays of towards those. Measured on practical metrics for filtering and correlating, our implementations are promising, while at the same time cutting the cost of deployment, according to the constraints. Consequently, we find it of practical relevance to consider methods, like ours, that are much easier and cheaper to deploy, compared to the existing ones.

Keywords

• Alert correlation
• alert filtering
• clustering
• intrusion detection system
• latent semantic analysis
• malware detection