Measuring Engagement in Provider-Guided Digital Health Interventions With a Conceptual and Analytical Framework Using Nurse WRITE as an Exemplar: Exploratory Study With an Iterative Approach

Abstract

BackgroundLimited guidance exists for analyzing participant engagement in provider-guided digital health interventions (DHIs). System usage is commonly assessed, with acknowledged limitations in measuring socio-affective and cognitive aspects of engagement. Nurse WRITE, an 8-week web-based nurse-guided DHI for managing symptoms among women with recurrent ovarian cancer, offers an opportunity to develop a framework for assessing multidimensional engagement. ObjectiveThis study aims to develop a conceptual and analytic framework to measure socio-affective, cognitive, and behavioral engagement with provider-guided DHIs. We then illustrate the framework’s ability to describe and categorize engagement using Nurse WRITE as an example. MethodsA sample of 68 participants from Nurse WRITE who posted on the message boards were included. We adapted a prior framework for conceptualizing and operationalizing engagement across 3 dimensions and finalized a set of 6 distinct measures. Using patients' posts, we created 2 socio-affective engagement measures—total count of socio-affective engagement classes (eg, sharing personal experience) and total word count—and 2 cognitive engagement measures—total count of cognitive engagement classes (eg, asking information-seeking questions) and average question completion percentage. Additionally, we devised behavioral engagement measures using website data—the total count of symptom care plans and plan reviews. k-Means clustering categorized the participants into distinct groups based on levels of engagement across 3 dimensions. Descriptive statistics and narratives were used to describe engagement in 3 dimensions. ResultsOn average, participants displayed socio-affective engagement 34.7 times, writing 14,851 words. They showed cognitive engagement 19.4 times, with an average of 78.3% completion of nurses' inquiries. Participants also submitted an average of 1.6 symptom care plans and 0.7 plan reviews. Participants were clustered into high (n=13), moderate (n=17), and low engagers (n=38) based on the 6 measures. High engagers wrote a median of 36,956 (IQR 26,199-46,265) words. They demonstrated socio-affective engagement approximately 81 times and cognitive engagement around 46 times, approximately 6 times that of the low engagers and twice that of the moderate engagers. High engagers had a median of 91.7% (IQR 82.2%-93.7%) completion of the nurses’ queries, whereas moderate engagers had 86.4% (IQR 80%-96.4%), and low engagers had 68.3% (IQR 60.1%-79.6%). High engagers completed a median of 3 symptom care plans and 2 reviews, while moderate engagers completed 2 plans and 1 review. Low engagers completed a median of 1 plan with no reviews. ConclusionsThis study developed and reported an engagement framework to guide behavioral intervention scientists in understanding and analyzing participants’ engagement with provider-guided DHIs. Significant variations in engagement levels across 3 dimensions highlight the importance of measuring engagement with provider-guided DHIs in socio-affective, cognitive, and behavioral dimensions. Future studies should validate the framework with other DHIs, explore the influence of patient and provider factors on engagement, and investigate how engagement influences intervention efficacy.