Quality assessment of diagnostic before-after studies: development of methodology in the context of a systematic review

BACKGROUND: Quality assessment tools for primary studies of test accuracy are relatively well developed, although only one is validated (QUADAS), but very little work has been done to develop tools to quality-assess studies evaluating the impact of diagnostic testing on management of patients (diagnostic or therapeutic yield). The recent draft NICE Guide to the Methods of Technology Appraisal (2007) suggests QUADAS "as a useful starting point for appraising studies that evaluate the sensitivity and specificity of a test" but does not mention how to quality assess diagnostic or therapeutic yield studies, in particular diagnostic before-after studies. In the context of undertaking a rapid systematic review of structural neuroimaging in psychosis for NICE, we describe the modifications that we made to QUADAS, our experience of this in practice and in relation to published theory on diagnostic or therapeutic yield studies. METHODS: The QUADAS tool was assessed for use in the review by two systematic reviewers with in-depth knowledge of the clinical area being reviewed and the types of studies being found in the searches that could answer the clinical question. Modifications were made following discussion as considered appropriate. RESULTS: Two QUADAS questions were removed altogether and. four additional questions were developed to capture additional quality issues not addressed by QUADAS. However, the developed checklist only partially helped to discern implications of the study designs on the results given. CONCLUSION: The division between topic-specific and more generic quality items of relevance to diagnostic before-after studies is important. With more time, further work could have been done to create a better quality assessment tool, for example by incorporating some of the issues mentioned in previous work in this area. This paper is a discussion around quality assessment and is intended to offer insights into the types of issues that should be assessed. A quality assessment tool for diagnostic before-after studies that incorporates items from QUADAS and published theory needs to be further developed and validated.