Elaborating on the assessment of the risk of bias in prognostic studies in pain rehabilitation using QUIPS—aspects of interrater agreement

BACKGROUND: Many studies have been performed to identify important prognostic factors for outcomes after rehabilitation of patients with chronic pain, and there is a need to synthesize them through systematic review. In this process, it is important to assess the study quality and risk of bias. The “Quality In Prognosis Studies” (QUIPS) tool has been developed for this purpose and consists of several prompting items categorized into six domains, and each domain is judged on a three-grade scale (low, moderate or high risk of bias). The aim of the present study was to determine the interrater agreement of the risk of bias assessment in prognostic studies of patients with chronic pain using QUIPS and to elaborate on the use of this instrument. METHODS: We performed a systematic review and a meta-analysis of prognostic factors for long-term outcomes after multidisciplinary rehabilitation in patients with chronic pain. Two researchers rated the risk of bias in 43 published papers in two rounds (15 and 28 papers, respectively). The interrater agreement and Cohen’s quadratic weighted kappa coefficient (κ) and 95% confidence interval (95%CI) were calculated in all domains and separately for the first and second rounds. RESULTS: The raters agreed in 61% of the domains (157 out of 258), with similar interrater agreement in the first (59%, 53/90) and second rounds (62%, 104/168). The overall weighted kappa coefficient (kappa for all domains and all papers) was weak: κ = 0.475 (95%CI = 0.358–0.601). A “minimal agreement” between the raters was found in the first round, κ = 0.323 (95%CI = 0.129–0.517), but increased to “weak agreement” in the second round, κ = 0.536 (95%CI = 0.390–0.682). CONCLUSION: Despite a relatively low interrater agreement, QUIPS proved to be a useful tool in assessing the risk of bias when performing a meta-analysis of prognostic studies in pain rehabilitation, since it demands of raters to discuss and investigate important aspects of study quality. Some items were particularly hard to differentiate in-between, and a learning phase was required to increase the interrater agreement. This paper highlights several aspects of the tool that should be kept in mind when rating the risk of bias in prognostic studies, and provides some suggestions on common pitfalls to avoid during this process. TRIAL REGISTRATION: PROSPERO CRD42016025339; registered 05 February 2016. ELECTRONIC SUPPLEMENTARY MATERIAL: The online version of this article (10.1186/s41512-019-0050-0) contains supplementary material, which is available to authorized users.