Testing the Use of Data Drawn from the Electronic Health Record to Compare Quality

INTRODUCTION: Health systems spend $1.5 billion annually reporting data on quality, but efficacy and utility for benchmarking are limited due, in part, to limitations of data sources. Our objective was to implement and evaluate measures of pediatric quality for three conditions using electronic health record (EHR)-derived data. METHODS: PCORnet networks standardized EHR-derived data to a common data model. In 13 health systems from 2 networks for 2015, we implemented the National Quality Forum measures: % children with sickle cell anemia who received a transcranial Doppler; % children on antipsychotics who had metabolic screening; and % pediatric acute otitis media with amoxicillin prescribed. Manual chart review assessed measure accuracy. RESULTS: Only 39% (N = 2,923) of 7,278 children on antipsychotics received metabolic screening (range: 20%–54%). If the measure indicated screening was performed, the chart agreed 88% of the time [95% confidence interval (CI): 81%–94%]; if it indicated screening was not done, the chart agreed 86% (95% CI: 78%–93%). Only 69% (N = 793) of 1,144 children received transcranial Doppler screening (range across sites: 49%–88%). If the measure indicated screening was performed, the chart agreed 98% of the time (95% CI: 94%–100%); if it indicated screening was not performed, the chart agreed 89% (95% CI: 82%–95%). For acute otitis media, chart review identified many qualifying cases missed by the National Quality Forum measure, which excluded a common diagnostic code. CONCLUSIONS: Measures of healthcare quality developed using EHR-derived data were valid and identified wide variation among network sites. This data can facilitate the identification and spread of best practices.