Updating a systematic review – what difference did it make? Case study of nicotine replacement therapy

AIMS: To examine the effect of updating a systematic review of nicotine replacement therapy on its contents and conclusions. METHODS: We examined the effects of regular updating of a systematic review of nicotine replacement therapy for smoking cessation. We considered two outcomes. First, we assessed the effect of adding new data to meta-analyses, comparing results in 2000 with the results in 1994. Second, we assessed qualitatively the ways inwhich the nature of the questions addressed by the review had changed between the two dates. For the first outcome, we compared the number of trials, the pooled estimate of effect using the odds ratio, and the results of pre-specified subgroup analyses, for nicotine gum and patch separately. Using a test for interaction, we assessed whether differences between estimates were statistically significant. RESULTS: There were ten new trials of nicotine gum between 1994 and 2000, and the meta-analytic effect changed little. For the nicotine patch the number of trials increased from 9 to 30, and the meta-analytic effect fell from 2.07 (95% CI 1.64 – 2.62) to 1.73 (95% CI 1.56 – 1.93). Apparent differences in relative effect in sub-groups found in 1994 were not found in 2000. The updated systematic review addressed a number of questions not identified in the original version. CONCLUSIONS: Updating the meta-analyses lead to a more precise estimate of the likely effect of the nicotine patch, but the clinical message was unchanged. Further placebo controlled NRT trials are not likely to add to the evidence base. It is questionable whether updating the meta-analyses to include them is worthwhile. The content of the systematic review has, however, changed, with the addition of data addressing questions not considered in the original review. There is a tension between the principle of identifying the important questions prior to conducting a review, and keeping the review up to date as primary research identifies new avenues of enquiry.