Variance estimation for effective coverage measures: A simulation study

BACKGROUND: Effective coverage research is increasing rapidly in global health and development, as researchers use a range of measures and combine data sources to adjust coverage for the quality of services received. However, most estimates of effective coverage that combine data sources are reported only as point estimates, which may be due to the challenge of calculating the variance for a composite measure. In this paper, we evaluate three methods to quantify the uncertainty in the estimation of effective coverage. METHODS: We conducted a simulation study to evaluate the performance of the exact, delta, and parametric bootstrap methods for constructing confidence intervals around point estimates that are calculated from combined data on coverage and quality. We assessed performance by computing the number of nominally 95% confidence intervals that contain the truth for a range of coverage and quality values and data source sample sizes. To illustrate these approaches, we applied the delta and exact methods to estimates of adjusted coverage of antenatal care (ANC) in Senegal. We used household survey data for coverage and health facility assessments for readiness to provide services. RESULTS: With small sample sizes, when the true effective coverage value was close to the boundaries 0 or 1, the exact and parametric bootstrap methods resulted in substantial over or undercoverage and, for the exact method, a high proportion of invalid confidence intervals, while the delta method yielded modest overcoverage. The proportion of confidence intervals containing the truth in all three methods approached the intended 95% with larger sample sizes and as the true effective coverage value moved away from the 0 or 1 boundary. Confidence intervals for adjusted ANC in Senegal were largely overlapping across the delta and exact methods, although at the sub-national level, the exact method produced invalid confidence intervals for estimates near 0 or 1. We provide the code to implement these methods. CONCLUSIONS: The uncertainty around an effective coverage estimate can be characterized; this should become standard practice if effective coverage estimates are to become part of national and global health monitoring. The delta method approach outperformed the other methods in this study; we recommend its use for appropriate inference from effective coverage estimates that combine data sources, particularly when either sample size is small. When used for estimates created from facility type or regional strata, these methods require assumptions of independence that must be considered in each example.