Using Cluster Analysis to Explore Engagement and e-Attainment as Emergent Behavior in Electronic Mental Health

BACKGROUND: In most e-mental health (eMH) research to date, adherence is defined according to a trial protocol. However, adherence to a study protocol may not completely capture a key aspect of why participants engage with eMH tools, namely, to achieve personal mental health goals. As a consequence, trial attrition reported as non-adherence or dropout may reflect e-attainment, the discontinuation of eMH engagement after personal goals have been met. Clarifying engagement patterns, such as e-attainment, and how these align with mental health trajectories, may help optimize eMH design and implementation science. OBJECTIVE: This study aimed to use clustering techniques to identify real-world engagement profiles in a community of eMH users and examine if such engagement profiles are associated with different mental health outcomes. The novelty of this approach was our attempt to identify actual user engagement behaviors, as opposed to employing engagement benchmarks derived from a trial protocol. The potential of this approach is to link naturalistic behaviors to beneficial mental health outcomes, which would be especially informative when designing eMH programs for the general public. METHODS: Between May 2013 and June 2018, Australian adults (N=43,631) signed up to myCompass, a self-guided eMH program designed to help alleviate mild to moderate symptoms of depression, anxiety, and stress. Recorded usage data included number of logins, frequency of mood tracking, number of started and completed learning activities, and number of tracking reminders set. A subset of users (n=168) completed optional self-assessment mental health questionnaires (Patient Health Questionnaire-9 item, PHQ-9; Generalized Anxiety Disorder Questionnaire-7 item, GAD-7) at registration and at 28 and 56 days after sign-up. Another subset of users (n=861) completed the PHQ-9 and GAD-7 at registration and at 28 days. RESULTS: Two-step cluster analyses revealed 3 distinct usage patterns across both subsamples: moderates, trackers, and super users, signifying differences both in the frequency of use as well as differences in preferences for program functionalities. For both subsamples, repeated measures analysis of variances showed significant decreases over time in PHQ-9 and GAD-7 scores. Time-by-cluster interactions, however, did not yield statistical significance in both subsamples, indicating that clusters did not predict symptom reduction over time. Interestingly, users who completed the self-assessment questionnaires twice had slightly but significantly lower depression and anxiety levels at sign-up compared with users who completed the questionnaires a third time at 56 days. CONCLUSIONS: Findings suggested that although users engaged with myCompass in different but measurable ways, those different usage patterns evoked equivalent mental health benefits. Furthermore, the randomized controlled trial paradigm may unintentionally limit the scope of eMH engagement research by mislabeling early mental health goal achievers as dropouts. More detailed and naturalistic approaches to study engagement with eMH technologies may improve program design and, ultimately, program effectiveness.