Validation studies for population-based intervention coverage indicators: design, analysis, and interpretation

BACKGROUND: Population-based intervention coverage indicators are widely used to track country and program progress in improving health and to evaluate health programs. Indicator validation studies that compare survey responses to a “gold standard” measure are useful to understand whether the indicator provides accurate information. The Improving Coverage Measurement (ICM) Core Group has developed and implemented a standard approach to validating coverage indicators measured in household surveys, described in this paper. METHODS: The general design of these studies includes measurement of true health status and intervention receipt (gold standard), followed by interviews with the individuals observed, and a comparison of the observations (gold standard) to the responses to survey questions. The gold standard should use a data source external to the respondent to document need for and receipt of an intervention. Most frequently, this is accomplished through direct observation of clinical care, and/or use of a study-trained clinician to obtain a gold standard diagnosis. Follow-up interviews with respondents should employ standard survey questions, where they exist, as well as alternative or additional questions that can be compared against the standard household survey questions. RESULTS: Indicator validation studies should report on participation at every stage, and provide data on reasons for non-participation. Metrics of individual validity (sensitivity, specificity, area under the receiver operating characteristic curve) and population-level validity (inflation factor) should be reported, as well as the percent of survey responses that are “don’t know” or missing. Associations between interviewer and participant characteristics and measures of validity should be assessed and reported. CONCLUSIONS: These methods allow respondent-reported coverage measures to be validated against more objective measures of need for and receipt of an intervention, and should be considered together with cognitive interviewing, discriminative validity, or reliability testing to inform decisions about which indicators to include in household surveys. Public health researchers should assess the evidence for validity of existing and proposed household survey coverage indicators and consider validation studies to fill evidence gaps.