Spending Carveouts Substantially Improve the Accuracy of Performance Measurement in Shared Savings Arrangements: Findings From Simulation Analysis of Medicaid ACOs

Accuracy of spending-based provider performance metrics is limited by random variation and components of spending that are uncontrollable by providers. Such components vary according to the care management focus and operational maturity of each provider group. This study uses data from New Jersey Medicaid accountable care organizations (ACOs) to examine how carving out uncontrollable components of spending affects the accuracy of performance measures in shared savings arrangements. Spending on injury care, custodial care in facilities (CCF), and amounts above $100 000 per patient are used as examples of potentially uncontrollable spending. Data from 7 applicant Medicaid ACOs are used to conduct Monte Carlo simulations examining the effects of carving out each type of uncontrollable spending under the assumption that controllable spending is reduced by 5%. The simulations show that failure to carve out uncontrollable injury care spending adds −3 to +1 percentage points of bias to the measurement of the true average savings rate (ASR) of 5% and can increase mean squared error (MSE) by a factor of up to 3. Failure to carve out uncontrollable CCF spending generates bias ranging from −4 to +9 percentage points and increases MSE by factors of 8 or more. Failure to carve out uncontrollable spending above $100 000 per person generates bias ranging from −5 to +5 percentage points and increases MSE by factors of 13 or more. Compared with the main modeling reported above, sensitivity analyses find even greater distortions in measured performance when uncontrollable spending is not carved out of the ASR calculation.