Totality of evidence of the effectiveness of repurposed therapies for COVID‐19: Can we use real‐world studies alongside randomized controlled trials?

Rapid and robust strategies to evaluate the efficacy and effectiveness of novel and existing pharmacotherapeutic interventions (repurposed treatments) in future pandemics are required. Observational “real‐world studies” (RWS) can report more quickly than randomized controlled trials (RCTs) and would have value were they to yield reliable results. Both RCTs and RWS were deployed during the coronavirus disease 2019 (COVID‐19) pandemic. Comparing results between them offers a unique opportunity to determine the potential value and contribution of each. A learning review of these parallel evidence channels in COVID‐19, based on quantitative modeling, can help improve speed and reliability in the evaluation of repurposed therapeutics in a future pandemic. Analysis of all‐cause mortality data from 249 observational RWS and RCTs across eight treatment regimens for COVID‐19 showed that RWS yield more heterogeneous results, and generally overestimate the effect size subsequently seen in RCTs. This is explained in part by a few study factors: the presence of RWS that are imbalanced for age, gender, and disease severity, and those reporting mortality at 2 weeks or less. Smaller studies of either type contributed negligibly. Analysis of evidence generated sequentially during the pandemic indicated that larger RCTs drive our ability to make conclusive decisions regarding clinical benefit of each treatment, with limited inference drawn from RWS. These results suggest that when evaluating therapies in future pandemics, (1) large RCTs, especially platform studies, be deployed early; (2) any RWS should be large and should have adequate matching of known confounders and long follow‐up; (3) reporting standards and data standards for primary endpoints, explanatory factors, and key subgroups should be improved; in addition, (4) appropriate incentives should be in place to enable access to patient‐level data; and (5) an overall aggregate view of all available results should be available at any given time.