A Review of Evaluations of Electronic Event-based Biosurveillance Systems

OBJECTIVE: To assess evaluations of electronic event-based biosurveillance systems (EEBS’s) and define priorities for EEBS evaluations. INTRODUCTION: EEBS’s that use near real-time information from the Internet are an increasingly important source of intelligence for public health organizations (1, 2). However, there has not been a systematic assessment of EEBS evaluations, which could identify uncertainties about current systems and guide EEBS development to effectively exploit digital information for surveillance. METHODS: We searched PubMed and consulted EEBS experts to identify EEBS’s that met the following criteria: uses publicly-available Internet info sources, includes events that impact humans, and has global scope. We constructed a list of 17 key evaluation variables using guidelines for evaluating health surveillance systems, and identified the key variables included in evaluations per EEBS, as well as the number of EEBS’s evaluated for each key variable (3,4). RESULTS: We identified 10 EEBS’s and 17 evaluations (Table 1). The number of evaluations per EEBS ranged from 1 (Gen-Db, GODsN) to 7 (GPHIN, HealthMap). The median number of variables assessed per EEBS was 6 (range, 3–12), with 5 (25%) evaluations assessing 7+ variables. Nine (53%) published evaluations contained quantitative assessments of at least 1 variable. The least-frequently studied variable was cost. No papers examined usefulness as specific public health decisions or outcomes resulting from early event detection, though 8 evaluations assessed usefulness by citing instances where the EEBS detected an outbreak earlier, or by eliciting user feedback. CONCLUSIONS: While EEBS’s have demonstrated their usefulness and accuracy for early outbreak detection, no evaluations have cited specific examples of public health decisions or outcomes resulting from the EEBS. Future evaluations should discuss these critical indicators of public health utility. They also should assess the novel aspects of EEBS and include variables such as policy readiness, system redundancy, input/output geography (5); and test the effects of combining EEBS’s into a “super system”.