Evaluating Study Design Rigor in Preclinical Cardiovascular Research: A Replication Study

BACKGROUND: Methodological rigor is a major priority in preclinical cardiovascular research to ensure experimental reproducibility and high quality research. Lack of reproducibility results in diminished translation of preclinical discoveries into medical practice and wastes resources. In addition, lack of reproducibility fosters uncertainty in the public’s acceptance of reported research results. METHODS: We evaluate the reporting of rigorous methodological practices in preclinical cardiovascular research studies published in leading scientific journals by screening articles for the inclusion of the following key study design elements (SDEs): considering sex as a biological variable, randomization, blinding, and sample size power estimation. We have specifically chosen to screen for these SDEs across articles pertaining to preclinical cardiovascular research studies published between 2011 and 2021. Our study replicates and extends a study published in 2017 by Ramirez et al. We hypothesized that there would be higher SDE inclusion across preclinical studies over time, that preclinical studies that also include human and animal substudies within the same study will exhibit greater SDE inclusion than animal-only preclinical studies, and that there will be a difference in SDE usage between large and small animal models. RESULTS: Overall, inclusion of SDEs was low. 15.2% of animal only studies included both sexes as a biological variable, 30.4% included randomization, 32.1% included blinding, and 8.2% included sample size estimation. Incorporation of SDE in preclinical studies did not significantly increase over the ten year time period in the articles we assessed. Although the inclusion of sex as a biological variable increased over the 10 year time frame, that change was not significant (p=0.411, corrected p=8.22). These trends were consistent across journals. Reporting of randomization and sample size estimation differs significantly between animal and human substudies (corrected p=3.690e-06 and corrected p=7.252e-08, respectively.) Large animal studies had a significantly greater percentage of blinding reported when compared to small animal studies (corrected p=0.01.) Additionally, overall, large animal studies tended to have higher SDE usage. CONCLUSIONS: In summary, evidence of methodological rigor varies substantially depending on the study type and model organisms used. Over the time period of 2011–2021, the reporting of SDEs within preclinical cardiovascular studies has not improved and suggests extensive evaluation of other SDEs used in cardiovascular research. Limited incorporation of SDEs within research hinders experimental reproducibility that is critical to future research.