Automated Surveillance of Outpatients with Pneumonia: A Performance Evaluation

OBJECTIVE: To determine if influenza surveillance should target all patients with acute respiratory infections (ARI) or only track pneumonia cases. INTRODUCTION: Effective responses to epidemics of infectious diseases hinge not only on early outbreak detection, but also on an assessment of disease severity. In recent work, we combined previously developed ARI case-detection algorithms (CDA) [1] with text analyses of chest imaging reports to identify ARI patients whose providers thought had pneumonia. In this work, we asked if a surveillance system aimed at patients with pneumonia would outperform one that monitors the full severity spectrum of ARI. METHODS: Time series of daily casecounts (backgrounds) were created by applying either an ARI CDA (ARI ICD-9 codeset [1]) or a Pneumonia CDA (ARI ICD-9 codes AND chest imaging obtained AND positive results from automated text analysis that identify those chest imaging reports that support the diagnosis of pneumonia) to electronic medical record (EMR) entries related to outpatient encounters at the VA Maryland Health Care System. We used an age-structured metapopulation influenza epidemic model for Baltimore to inject factitious influenza cases into backgrounds. Injections were discounted by the known sensitivity of the ARI CDA [1]. For injections into the pneumonia backgrounds time series, factitious ARI cases were further discounted by the expected pneumonia rate in the modeled influenza epidemic (10%). From the time of injection, EARS or CUSUM statistics [2,3] were applied on each successive day on paired back-ground+injection vs. background-only time series. Each injection-prospective-surveillance cycle was repeated 52 times, each time with the injection shifted to a different week of the one-year study period (2010–11). We computed: 1) the “Detection Delay”, the average time from injection to the first alarm present in the back-ground+injection dataset but absent from the background-only dataset; 2) the “False-Alarm Rate” (FAR), defined as the number of unique false-alarms originating in the background-only dataset during the study year, divided by 365 days. To create activity monitoring operating characteristic (AMOC) curves, we empirically determined the corresponding Delay-FAR pairs over a wide range of alarm thresholds. RESULTS: The Figure compares AMOC curves for otherwise identical surveillance systems that included either any ARI outpatient visits (red circles, using the EARS W2c statistic [2] and blue triangles, using the CUSUM statistic) or pneumonia (blue triangles, using the CUSUM statistic modified for sparse data [3]). Note that Detection Delay (y-axis) is lower at any given FAR when surveillance aims at patients with pneumonia. Sensitivity analysis suggests that this advantage remains true when pneumonia complicates influenza ≥ 5% of the time. CONCLUSIONS: Our results suggest that EMR-based influenza surveillance that targets patients with pneumonia can outperform systems that monitor all ARI patients. [Figure: see text]