The orthopaedic trauma literature: an evaluation of statistically significant findings in orthopaedic trauma randomized trials

BACKGROUND: Evidence-based medicine posits that health care research is founded upon clinically important differences in patient centered outcomes. Statistically significant differences between two treatments may not necessarily reflect a clinically important difference. We aimed to quantify the sample sizes and magnitude of treatment effects in a review of orthopaedic randomized trials with statistically significant findings. METHODS: We conducted a comprehensive search (PubMed, Cochrane) for all randomized controlled trials between 1/1/95 to 12/31/04. Eligible studies include those that focused upon orthopaedic trauma. Baseline characteristics and treatment effects were abstracted by two reviewers. Briefly, for continuous outcome measures (ie functional scores), we calculated effect sizes (mean difference/standard deviation). Dichotomous variables (ie infection, nonunion) were summarized as absolute risk differences and relative risk reductions (RRR). Effect sizes >0.80 and RRRs>50% were defined as large effects. Using regression analysis we examined the association between the total number of outcome events and treatment effect (dichotomous outcomes). RESULTS: Our search yielded 433 randomized controlled trials (RCTs), of which 76 RCTs with statistically significant findings on 184 outcomes (122 continuous/62 dichotomous outcomes) met study eligibility criteria. The mean effect size across studies with continuous outcome variables was 1.7 (95% confidence interval: 1.43–1.97). For dichotomous outcomes, the mean risk difference was 30% (95%confidence interval:24%–36%) and the mean relative risk reduction was 61% (95% confidence interval: 55%–66%; range: 0%–97%). Fewer numbers of total outcome events in studies was strongly correlated with increasing magnitude of the treatment effect (Pearson's R = -0.70, p < 0.01). When adjusted for sample size, the number of outcome events revealed an independent association with the size of the treatment effect (Odds ratio = 50, 95% confidence interval: 3.0–1000, p = 0.006). CONCLUSION: Our review suggests that statistically significant results in orthopaedic trials have the following implications-1) On average large risk reductions are reported 2) Large treatment effects (>50% relative risk reduction) are correlated with few number of total outcome events. Readers should interpret the results of such small trials with these issues in mind.