Let’s decide what would be convincing, conduct randomized trials with rigorous comparison conditions, and report tests of moderation and publication bias in meta-analyses

We appreciate Jacobson and colleagues’ thoughtful commentary on our meta-review of mobile phone-based interventions for mental health. In this response, we address 2 issues raised: requiring low to moderate heterogeneity (I(2) < 50%) and requiring no evidence of publication bias for evidence to be classified as “convincing.” While we agree these represent a high bar, we disagree that these requirements are destined to fail. Other effect sizes reported in the literature, including effect sizes related to mental health interventions and effect sizes related to mobile health (mHealth) interventions (although not their combination) have met requirements for convincing evidence. Jacobson and colleagues argue that features of the mHealth interventions may produce heterogeneity when meta-analyses combine across intervention types. However, several of the effect sizes we reviewed were based on relatively homogeneous portions of the literature and many of the effect sizes we reviewed showed low to moderate heterogeneity. Ideally, future meta-analyses will examine intervention features as moderators of treatment effects. While an absence of publication bias may be a stringent criterion, all but 2 of the 34 effect sizes we reviewed did not report formal tests of publication bias. Clearly, there is a need to reach consensus on how the strength of evidence for mHealth interventions can be evaluated. From our perspective, convincing evidence will ultimately come from large-scale randomized controlled trials employing rigorous comparison conditions along with meta-analyses that do not combine across control condition types, that examine theoretically important moderators, and report formal tests of publication bias. It is this kind of evidence that the public, the clinicians, and the scientific community may need to encourage adoption of mHealth interventions for mental health treatment and prevention.