Challenges in Estimating the Impact of Vaccination with Sparse Data

BACKGROUND: The synthetic control model is a powerful tool to quantify the population-level impact of vaccines because it can adjust for trends unrelated to vaccination using a composite of control diseases. Because vaccine impact studies are often conducted using smaller, subnational datasets, we evaluated the performance of synthetic control models with sparse time series data. To obtain more robust estimates of vaccine impacts from noisy time series, we proposed a possible alternative approach, STL+PCA method (seasonal-trend decomposition plus principal component analysis), which first extracts smoothed trends from the control time series and uses them to adjust the outcome. METHODS: Using both the synthetic control and STL+PCA models, we estimated the impact of 10-valent pneumococcal conjugate vaccine on pneumonia hospitalizations among cases <12 months and 80+ years of age during 2004–2014 at the subnational level in Brazil. We compared the performance of these models using simulation analyses. RESULTS: The synthetic control model was able to adjust for trends unrelated to 10-valent pneumococcal conjugate vaccine in larger states but not in smaller states. Simulation analyses showed that the estimates obtained with the synthetic control approach were biased when there were fewer cases, and only 4% of simulations had credible intervals covering the true estimate. In contrast, the STL+PCA analysis had 90% lower bias and had 95% of simulations, with credible intervals covering the true estimate. CONCLUSIONS: Estimates from the synthetic control model might be biased when data are sparse. The STL+PCA model provides more accurate evaluations of vaccine impact in smaller populations.