The risk of bias in randomized controlled trials in otorhinolaryngology: hardly any improvement since 1950

BACKGROUND: Randomized Controlled Trials (RCTs) represent the most valuable study design to evaluate the effectiveness of therapeutic interventions. However, flaws in design, conduct, analysis, and reporting of RCTs can cause the effect of an intervention to be under- or overestimated. These biased RCTs may be included in literature reviews. To make the assessment of Risk of Bias (RoB) consistent and transparent, Cochrane published a RoB tool, with which RoB is assessed per item as “low”, “unclear” or “high”. Our objective was to provide an overview of RoB assessments of RCTs in otorhinolaryngology over time, and to identify items where improvement is still warranted. METHODS: We retrieved Cochrane reviews in the otorhinolaryngologic research field published in 2012 and 2013. We used all judgments per item as assessed by the review authors of the included RCTs. We evaluated the association between “low RoB” vs. “unclear and high RoB” and the year of publication (time strata: ‘<1990’, ‘1990–1995’, ‘1996–2000’, ‘2001–2005’, ‘2006–2012’) per item using binary logistic regression. RESULTS: We extracted the RoB assessments from 42 Cochrane reviews that had included 402 RCTs (median number of RCTs per review: 7, range 1–40). In total 2,356 items were assessed (mean number of assessed items per RCT: 5.9, standard deviation 1.8). On binary logistic regression, RCTs published in 2006–2012, compared with those published before 1990, were more likely to have a low RoB for two items: random sequence generation (odds ratio 6.09 [95% confidence interval: 3.11–11.95]) and allocation concealment (3.59 [1.87–6.90]). On all other items, there was no significant increase in the proportion of low RoB when comparing RCTs published in 2006–2012 with RCTs published before 1990. CONCLUSION: Although there were some positive developments in the RoB assessments in otorhinolaryngology, a further decrease in RoB is still warranted on several items. Currently, biased RCTs are included in Cochrane reviews and effects of therapeutic interventions can be under- or overestimated, with implications for clinical patient care. ELECTRONIC SUPPLEMENTARY MATERIAL: The online version of this article (doi:10.1186/s12901-017-0036-x) contains supplementary material, which is available to authorized users.