Predicting Early Dropout in a Digital Tobacco Cessation Intervention: Replication and Extension Study

Abstract

BackgroundDetecting early dropout from digital interventions is crucial for developing strategies to enhance user retention and improve health-related behavioral outcomes. Bricker and colleagues proposed a single metric that accurately predicted early dropout from 4 digital tobacco cessation interventions based on log-in data in the initial week after registration. Generalization of this method to additional interventions and modalities would strengthen confidence in the approach and facilitate additional research drawing on it to increase user retention. ObjectiveThis study had two research questions (RQ): RQ1—can the study by Bricker and colleagues be replicated using data from a large-scale observational, multimodal intervention to predict early dropout? and RQ2—can first-week engagement patterns identify users at the greatest risk for early dropout, to inform development of potential “rescue” interventions? MethodsData from web users were drawn from EX, a freely available, multimodal digital intervention for tobacco cessation (N=70,265). First-week engagement was operationalized as any website page views or SMS text message responses within 1 week after registration. Early dropout was defined as having no subsequent engagement after that initial week through 1 year. First, a multivariate regression model was used to predict early dropout. Model predictors were dichotomous measures of engagement in each of the initial 6 days (days 2-7) following registration (day 1). Next, 6 univariate regression models were compared in terms of their discrimination ability to predict early dropout. The sole predictor of each model was a dichotomous measure of whether users had reengaged with the intervention by a particular day of the first week (calculated separately for each of 2-7 days). ResultsFor RQ1, the area under the receiver operating characteristic curve (AUC) of the multivariate model in predicting dropout after 1 week was 0.72 (95% CI 0.71-0.73), which was within the range of AUC metrics found in the study by Bricker and colleagues. For RQ2, the AUCs of the univariate models increased with each successive day until day 4 (0.66, 95% CI 0.65-0.67). The sensitivity of the models decreased (range 0.79-0.59) and the specificity increased (range 0.48-0.73) with each successive day. ConclusionsThis study provides independent validation of the use of first-week engagement to predict early dropout, demonstrating that the method generalizes across intervention modalities and engagement metrics. As digital intervention researchers continue to address the challenges of low engagement and early dropout, these results suggest that first-week engagement is a useful construct with predictive validity that is robust across interventions and definitions. Future research should explore the applicability and efficiency of this model to develop interventions to increase retention and improve health behavioral outcomes.