Outcome choice and definition in systematic reviews leads to few eligible studies included in meta-analyses: a case study

BACKGROUND: There is broad recognition of the importance of evidence in informing clinical decisions. When information from all studies included in a systematic review (“review”) does not contribute to a meta-analysis, decision-makers can be frustrated. Our objectives were to use the field of eyes and vision as a case study and examine the extent to which authors of Cochrane reviews conducted meta-analyses for their review’s pre-specified main outcome domain and the reasons that some otherwise eligible studies were not incorporated into meta-analyses. METHODS: We examined all completed systematic reviews published by Cochrane Eyes and Vision, as of August 11, 2017. We extracted information about each review’s outcomes and, using an algorithm, categorized one outcome as its “main” outcome. We calculated the percentage of included studies incorporated into meta-analyses for any outcome and for the main outcome. We examined reasons for non-inclusion of studies into the meta-analysis for the main outcome. RESULTS: We identified 175 completed reviews, of which 125 reviews included two or more studies. Across these 125 reviews, the median proportions of studies incorporated into at least one meta-analysis for any outcome and for the main outcome were 74% (interquartile range [IQR] 0–100%) and 28% (IQR 0–71%), respectively. Fifty-one reviews (41%) could not conduct a meta-analysis for the main outcome, mostly because fewer than two included studies measured the outcome (21/51 reviews) or the specific measurements for the outcome were inconsistent (16/51 reviews). CONCLUSIONS: Outcome choice during systematic reviews can lead to few eligible studies included in meta-analyses. Core outcome sets and improved reporting of outcomes can help solve some of these problems.