Evaluation of Machine Learning to Detect Influenza Using Wearable Sensor Data and Patient-Reported Symptoms: Cohort Study

Abstract

BackgroundMachine learning offers quantitative pattern recognition analysis of wearable device data and has the potential to detect illness onset and monitor influenza-like illness (ILI) in patients who are infected. ObjectiveThis study aims to evaluate the ability of machine-learning algorithms to distinguish between participants who are influenza positive and influenza negative in a cohort of symptomatic patients with ILI using wearable sensor (activity) data and self-reported symptom data during the latent and early symptomatic periods of ILI. MethodsThis prospective observational cohort study used the extreme gradient boosting (XGBoost) classifier to determine whether a participant was influenza positive or negative based on 3 models using symptom-only data, activity-only data, and combined symptom and activity data. Data were collected from the Home Testing of Respiratory Illness (HTRI) study and FluStudy2020, both conducted between December 2019 and October 2020. The model was developed using the FluStudy2020 data and tested on the HTRI data. Analyses included participants in these studies with an at-home influenza diagnostic test result. Fitbit (Google LLC) devices were used to measure participants’ steps, heart rate, and sleep parameters. Participants detailed their ILI symptoms, health care–seeking behaviors, and quality of life. Model performance was assessed by area under the curve (AUC), balanced accuracy, recall (sensitivity), specificity, precision (positive predictive value), negative predictive value, and weighted harmonic mean of precision and recall (F2) score. ResultsAn influenza diagnostic test result was available for 953 and 925 participants in HTRI and FluStudy2020, respectively, of whom 848 (89%) and 840 (90.8%) had activity data. For the training and validation sets, the highest performing model was trained on the combined symptom and activity data (training AUC=0.77; validation AUC=0.74) versus symptom-only (training AUC=0.73; validation AUC=0.72) and activity-only (training AUC=0.68; validation AUC=0.65) data. For the FluStudy2020 test set, the performance of the model trained on combined symptom and activity data was closely aligned with that of the symptom-only model (combined symptom and activity test AUC=0.74; symptom-only test AUC=0.74). These results were validated using independent HTRI data (combined symptom and activity evaluation AUC=0.75; symptom-only evaluation AUC=0.74). The top features guiding influenza detection were cough; mean resting heart rate during main sleep; fever; total minutes in bed for the combined model; and fever, cough, and sore throat for the symptom-only model. ConclusionsMachine-learning algorithms had moderate accuracy in detecting influenza, suggesting that previous findings from research-grade sensors tested in highly controlled experimental settings may not easily translate to scalable commercial-grade sensors. In the future, more advanced wearable sensors may improve their performance in the early detection and discrimination of viral respiratory infections.