Just how plain are plain tobacco packs: re-analysis of a systematic review using multilevel meta-analysis suggests lessons about the comparative benefits of synthesis methods

BACKGROUND: Comparisons between narrative synthesis and meta-analysis as synthesis methods in systematic reviews are uncommon within the same systematic review. We re-analysed a systematic review on the effects of plain packaging of tobacco on attractiveness. We sought to compare different synthesis approaches within the same systematic review and shed light on the comparative benefits of each approach. METHODS: In our re-analysis, we included results relating to attractiveness in included reports. We extracted findings from studies and converted all estimates of differences in attractiveness to Cohen’s d. We used multilevel meta-analysis to account for clustering of effect sizes within studies. RESULTS: Of the 19 studies reporting results on attractiveness, seven studies that included between-subjects analyses could be included in the meta-analysis. Plain packs were less attractive than branded packs (d = − 0.59, 95% CI [− 0.71, − 0.47]), with negligible but uncertain between-studies heterogeneity (I(2) = 0%, 95% CI [0.00, 70.81]) and high within-study heterogeneity (I(2) = 92.6%, 95% CI [91.04, 93.90]). CONCLUSIONS: The meta-analysis found, similar to the narrative synthesis, that respondents typically rated plain packaging as less attractive than alternative (e.g. branded) tobacco packs. However, there were several trade-offs between analysis methods in the types and bodies of evidence each one contained and in the difference between partial precision and breadth of conclusions. Analysis methods were different in respect of the role of judgement and contextual variation and in terms of estimation and unexpected effect modification. In addition, we noted that analysis methods were different in how they accounted for heterogeneity and consistency. ELECTRONIC SUPPLEMENTARY MATERIAL: The online version of this article (10.1186/s13643-018-0821-7) contains supplementary material, which is available to authorized users.