Evaluating health policies with subnational disparities: a text-mining analysis of the Urban Employee Basic Medical Insurance Scheme in China

Subnational disparities in most health systems often defy ‘one-size-fits-all’ approach in policy implementation. When local authorities implement a national policy in a decentralized context, they behave as a strategic policy actor in specifying the central mandates, selecting appropriate tools and setting key implementation parameters. Local policy discretion leads to diverse policy mixes across regions, thus complicating evidence-based evaluations of policy impacts. When measuring complex policy reforms, mainstream policy evaluation methodologies have tended to adopt simplified policy proxies that often disguise distinct policy choices across localities, leaving the heterogeneous effects of the same generic policy largely unknown. Using the emerging ‘text-as-data’ methodology and drawing from subnational policy documents, this study developed a novel approach to policy measurement through analysing policy big data. We applied this approach to examine the impacts of China’s Urban Employee Basic Medical Insurance (UEBMI) on individuals’ out-of-pocket (OOP) spending. We found substantial disparities in policy choices across prefectures when categorizing the UEBMI policy framework into benefit-expansion and cost-containment reforms. Overall, the UEBMI policies lowered enrollees’ OOP spending in prefectures that embraced both benefit-expansion and cost-containment reforms. In contrast, the policies produced ill effects on OOP spending of UEBMI enrollees and uninsured workers in prefectures that carried out only benefit-expansion or cost-containment reforms. The micro-level impacts of UEBMI enrolment on OOP spending were conditional on whether prefectural benefit-expansion and cost-containment reforms were undertaken in concert. Only in prefectures that promulgated both types of reforms did UEBMI enrolment reduce OOP spending. These findings contribute to a comprehensive text-mining measurement approach to locally diverse policy efforts and an integration of macro-level policy analysis and micro-level individual analysis. Contextualizing policy measurements would improve the methodological rigour of health policy evaluations. This paper concludes with implications for health policymakers in China and beyond.