ATQ: alert time quality, an evaluation metric for assessing timely epidemic detection models within a school absenteeism-based surveillance system

Abstract

Abstract Background Wellington-Dufferin-Guelph Public Health (WDGPH) has conducted an absenteeism-based influenza surveillance program in the WDG region of Ontario, Canada since 2008, using a 10% absenteeism threshold to raise an alert for the implementation of mitigating measures. A recent study indicated that model-based alternatives, such as distributed lag seasonal logistic regression models, provided improved alerts for detecting an upcoming epidemic. However model evaluation and selection was primarily based on alert accuracy, measured by the false alert rate (FAR), and failed to optimize timeliness. Here, a new metric that simultaneously evaluates epidemic alert accuracy and timeliness is proposed. The alert time quality (ATQ) metric is investigated as a model selection criterion on both a simulated and real data set. Methods The ATQ assessed alerts on a gradient, where alerts raised incrementally before or after an optimal day were considered informative, but were penalized for lack of timeliness. Summary statistics of ATQ, average alert time quality (AATQ) and first alert time quality (FATQ), were used for model evaluation and selection. Alerts raised by ATQ and FAR selected models were compared. Daily elementary school absenteeism and laboratory-confirmed influenza case data collected by WDGPH were used for demonstration and evaluation of the proposed metric. A simulation study that mimicked the WDG population and influenza demographics was conducted for further evaluation of the proposed metric. Results The FATQ-selected model raised acceptable first alerts most frequently, while the AATQ-selected model raised first alerts within the ideal range most frequently. Conclusions Models selected by either FATQ or AATQ would more effectively predict community influenza activity with the local community than those selected by FAR.

Keywords

• Absenteeism surveillance system
• Influenza
• Epidemic detection
• Simulation study
• Evaluation metric