Data-driven human and bot recognition from web activity logs based on hybrid learning techniques

Abstract

Distinguishing between web traffic generated by bots and humans is an important task in the evaluation of online marketing campaigns. One of the main challenges is related to only partial availability of the performance metrics: although some users can be unambiguously classified as bots, the correct label is uncertain in many cases. This calls for the use of classifiers capable of explaining their decisions. This paper demonstrates two such mechanisms based on features carefully engineered from web logs. The first is a man-made rule-based system. The second is a hierarchical model that first performs clustering and next classification using human-centred, interpretable methods. The stability of the proposed methods is analyzed and a minimal set of features that convey the class-discriminating information is selected. The proposed data processing and analysis methodology are successfully applied to real-world data sets from online publishers.

Keywords

• Web logs
• Classification
• Clustering
• Web traffic
• Bots
• Interpretability