Greedy caliper propensity score matching can yield variable estimates of the treatment‐outcome association—A simulation study

PURPOSE: Greedy caliper propensity score (PS) matching is dependent on randomness, which can ultimately affect causal estimates. We sought to investigate the variation introduced by this randomness. METHODS: Based on a literature search to define the simulation parameters, we simulated 36 cohorts of different sizes, treatment prevalence, outcome prevalence, treatment‐outcome‐association. We performed 1:1 caliper and nearest neighbor (NN) caliper PS‐matching and repeated this 1000 times in the same cohort, before calculating the treatment‐outcome association. RESULTS: Repeating caliper and NN caliper matching in the same cohort yielded large variations in effect estimates, in all 36 scenarios, with both types of matching. The largest variation was found in smaller cohorts, where the odds ratio (OR) ranged from 0.53 to 10.00 (IQR of ORs: 1.11‐1.67). The 95% confidence interval was not consistently overlapping a neutral association after repeating the matching with both algorithms. We confirmed these findings in a noninterventional example study. CONCLUSION: Caliper PS‐matching can yield highly variable estimates of the treatment‐outcome association if the analysis is repeated.