Reliability of a patient survey assessing cost-related changes in health care use among high deductible health plan enrollees

BACKGROUND: Recent increases in patient cost-sharing for health care have lent increasing importance to monitoring cost-related changes in health care use. Despite the widespread use of survey questions to measure changes in health care use and related behaviors, scant data exists on the reliability of such questions. METHODS: We administered a cross-sectional survey to a stratified random sample of families in a New England health plan's high deductible health plan (HDHP) with ≥ $500 in annualized out-of-pocket expenditures. Enrollees were asked about their knowledge of their plan, information seeking, behavior change associated with having a deductible, experience of delay in care due in part to cost, and hypothetical delay in care due in part to cost. Initial respondents were mailed a follow-up survey within two weeks of each family returning the original survey. We computed several agreement statistics to measure the test-retest reliability for select questions. We also conducted continuity adjusted chi-square, and McNemar tests in both the original and follow-up samples to measure the degree to which our results could be reproduced. Analyses were stratified by self-reported income. RESULTS: The test-retest reliability was moderate for the majority of questions (0.41 - 0.60) and the level of test-retest reliability did not differ substantially across each of the broader domains of questions. The observed proportions of respondents with delayed or foregone pediatric, adult, or any family care were similar when comparing the original and follow-up surveys. In the original survey, respondents in the lower-income group were more likely to delay or forego pediatric care, adult care, or any family care. All of the tests comparing income groups in the follow-up survey produced the same result as in the original survey. CONCLUSIONS: In this population of HDHP beneficiaries, we found that survey questions concerning plan knowledge, information seeking, and delayed or foregone care were moderately reliable. Our results offer reassurance for researchers using survey information to study the effects cost sharing on health care utilization.