Meta-analysis of clinical trials in the 2020s and beyond: a paradigm shift needed

BACKGROUND: A peer-reviewed meta-analysis methods article mathematically proved that mainstream random-effects methods, “weights inversely proportional to the estimated variance,” are flawed and can lead to faulty public health recommendations. Because the arguments causing this off-label (unproven) use of mainstream practices were subtle, changing these practices will require much clearer explanations that can be grasped by clinical and translational scientists. There are five assumptions underlying the mainstream’s derivation of its statistical properties. This paper will demonstrate that if the first is true, it follows that the last two are false. Ratio estimation, borrowed from classical survey sampling, provides a rigorous alternative. Papers reporting results rarely fully disclose these assumptions. This is analogous to watching TV ads with the sound muted. You see high quality of life and do not hear about the complications. This article is a poster child for translational science, as it takes a theoretical discovery from the biostatistical world, translates it into language clinical scientists can understand, and thereby can change their research practice. AIM: This article is aimed at future applications of meta-analysis of complete collections of randomized clinical trials. It leaves it to past authors as to whether to reanalyze their data. No blame for past use is assessed. METHODS: By treating the individual completed studies in the meta-analysis as a random sample from a conceptual universe of completed studies, we use ratio estimation to obtain estimates of relative risk (ratio of failure rates treatment: control) and mean differences, projecting our sample value to estimate the universe’s value. RESULTS: Two examples demonstrate that the mainstream methods likely adversely impacted major treatment options. A third example shows that the key mainstream presumption of independence between the study weights and study estimates cannot be supported. CONCLUSION: There is no rationale for ever using the mainstream for meta-analysis of randomized clinical trials. RELEVANCE FOR PATIENTS: Future meta-analysis of clinical trials should never employ mainstream methods. Doing so could lead to potentially harmful public health policy recommendations. Clinical researchers need to play a primary role to assure good research practices in meta-analysis.