Challenges and lessons learned from using anchoring vignettes to explore quality of life response behavior

PURPOSE: Asking patients to rate health-related quality of life (HRQoL) of hypothetical individuals described in anchoring vignettes has been proposed to enhance knowledge on how patients understand and respond to HRQoL questionnaires. In this article, we describe the development of anchoring vignettes and explore their utility for measuring response shift in patients’ self-reports of HRQoL. METHODS: We conducted an explorative mixed-methods study. One hundred patients with multiple sclerosis or psoriasis participated in two interviews at intervals of 3–6 months. During both interviews, patients assessed HRQoL of 16 hypothetical individuals on the SF-12 questionnaire (two vignettes for each of the eight domains of the SF-12). In addition to these quantitative ratings, we used the think-aloud method to explore changes in patients’ verbalization of their decision processes during vignette ratings. RESULTS: Agreement of vignette ratings at baseline and follow-up was low (ICCs < 0.55). In addition, paired sample t-tests revealed no significant directional mean changes in vignette ratings. Thus, ratings changed non-directionally, neither confirming retest reliability nor a systematic change of assessment. Furthermore, patients’ verbalization of their decision processes did not indicate whether or not the assessment strategy of individual patients had changed. CONCLUSIONS: Patients’ ratings of anchoring vignettes fluctuate non-directionally over time. The think-aloud method appears not to be informative in exploring whether these fluctuations are due to changes in the individual decision process. Overall, vignettes might not be an appropriate approach to explore response shift, at least with regard to the specific target population and the use of the SF-12. ELECTRONIC SUPPLEMENTARY MATERIAL: The online version of this article (10.1007/s11136-020-02488-4) contains supplementary material, which is available to authorized users.