Monitoring events with application to syndromic surveillance using social media data

Abstract

Abstract Availability of time series data in different domains has resulted in approaches for outbreak detection. A popular alternative to detect outbreaks is to use daily counts of events. However, time between events (TBE) has proven to be a useful alternative, especially in the case of sudden, unexpected events. Past work that uses TBE for monitoring events assumes that the in‐control number of events is up to 10 per day. In this article, we derive robust monitoring plans that are scalable when the in‐control counts are higher than 10 per day but less than 100 per counting period (eg, day). TBE values are generally nonhomogeneous across days and within days. This makes the volume of data to train the technology a challenge, and this challenge increases the volume of data needed to design the charts. This article discusses these challenges and suggests solutions for data that are known to be Weibull‐distributed. We present our results in two parts. The first is a simulated dataset that controls parameters of the plan such as the daily counts of events. We then show how the monitoring plans can be applied to the detection of syndromes (ie, disease outbreaks) using social media data.

Keywords

• counting processes
• EWMA
• statistical process control
• time between events