Do Differential Response Rates to Patient Surveys Between Organizations Lead to Unfair Performance Comparisons?: Evidence From the English Cancer Patient Experience Survey

BACKGROUND: Patient surveys typically have variable response rates between organizations, leading to concerns that such differences may affect the validity of performance comparisons. OBJECTIVE: To explore the size and likely sources of associations between hospital-level survey response rates and patient experience. RESEARCH DESIGN, SUBJECTS, AND MEASURES: Cross-sectional mail survey including 60 patient experience items sent to 101,771 cancer survivors recently treated by 158 English NHS hospitals. Age, sex, race/ethnicity, socioeconomic status, clinical diagnosis, hospital type, and region were available for respondents and nonrespondents. RESULTS: The overall response rate was 67% (range, 39% to 77% between hospitals). Hospitals with higher response rates had higher scores for all items (Spearman correlation range, 0.03–0.44), particularly questions regarding hospital-level administrative processes, for example, procedure cancellations or medical note availability. From multivariable analysis, associations between individual patient experience and hospital-level response rates were statistically significant (P<0.05) for 53/59 analyzed questions, decreasing to 37/59 after adjusting for case-mix, and 25/59 after further adjusting for hospital-level characteristics. Predicting responses of nonrespondents, and re-estimating hypothetical hospital scores assuming a 100% response rate, we found that currently low performing hospitals would have attained even lower scores. Overall nationwide attainment would have decreased slightly to that currently observed. CONCLUSIONS: Higher response rate hospitals have more positive experience scores, and this is only partly explained by patient case-mix. High response rates may be a marker of efficient hospital administration, and higher quality that should not, therefore, be adjusted away in public reporting. Although nonresponse may result in slightly overestimating overall national levels of performance, it does not appear to meaningfully bias comparisons of case-mix-adjusted hospital results.