Statistical Decision Properties of Imprecise Trials Assessing Coronavirus Disease 2019 (COVID-19) Drugs

OBJECTIVES: Researchers studying treatment of coronavirus disease 2019 (COVID-19) have reported findings of randomized trials comparing standard care with care augmented by experimental drugs. Many trials have small sample sizes, so estimates of treatment effects are imprecise. Hence, clinicians may find it difficult to decide when to treat patients with experimental drugs. A conventional practice when comparing standard care and an innovation is to choose the innovation only if the estimated treatment effect is positive and statistically significant. This practice defers to standard care as the status quo. We study treatment choice from the perspective of statistical decision theory, which considers treatment options symmetrically when assessing trial findings. METHODS: We use the concept of near-optimality to evaluate criteria for treatment choice. This concept jointly considers the probability and magnitude of decision errors. An appealing criterion from this perspective is the empirical success rule, which chooses the treatment with the highest observed average patient outcome in the trial. RESULTS: Considering the design of some COVID-19 trials, we show that the empirical success rule yields treatment choices that are much closer to optimal than those generated by prevailing decision criteria based on hypothesis tests. CONCLUSION: Using trial findings to make near-optimal treatment choices rather than perform hypothesis tests should improve clinical decision making.