Detecting Changes in Chief Complaint Word Count: Effects on Syndromic Surveillance

OBJECTIVE: To identify changes in emergency department (ED) syndromic surveillance data by analyzing trends in chief complaint (CC) word count; to compare these changes to coding changes reported by EDs; and to examine how these changes might affect the ability of syndromic surveillance systems to identify syndromes in a consistent manner. INTRODUCTION: The New York City (NYC) Department of Health and Mental Hygiene (DOHMH) receives daily ED data from 49 of NYC’s 52 hospitals, representing approximately 95% of ED visits citywide. Chief complaint (CC) is categorized into syndrome groupings using text recognition of symptom key-words and phrases. Hospitals are not required to notify the DOHMH of any changes to procedures or health information systems (HIS). Previous work noticed that CC word count varied over time within and among EDs. The variations seen in CC word count may affect the quality and type of data received by the DOHMH, thereby affecting the ability to detect syndrome visits consistently. METHODS: The daily mean number of words in the chief complaint field were examined by hospital from 2008–2011. Spectral analyses were performed on daily CC word count by hospital to explore temporal trends. Change Point Analysis (CPA) using Taylor’s Method with a maximum change level of four was conducted on the CC field by hospital using 1,000 bootstrap samples. According to Taylor, a level one change is the most important change detected on the program’s first pass through the data. For this analysis, a change point was considered significant if it was level 1, detected an average change of more than 0.50 words per day, and was sustained for at least 6 months before a level 2 change of at least 0.50 words occurred. Results of the CPA were compared to reported changes identified by a survey conducted by DOHMH staff of 49 hospitals that collected information about their HIS and coding practices, including any recent system changes. When a significant level one change was identified, time series graphs for six months before and after the change were created for five syndromes (cold, diarrhea, fever-flu, influenza-like-illness, and respiratory) and the syndrome’s constituent symptom categories (e.g. cough fever, etc.). Changes in syndrome count and composition at the level one change in word count were noted. RESULTS: The mean chief complaint word count across all hospitals from 2008 – 2011 in NYC was 3.14, with a range of 0 to 18 words. CPA detected a significant level 1 change in 21 hospitals, with a mean change of 0.60 words, with 9 increases (mean= 0.71 words) and 12 decreases (mean= 0.53 words). According to the results of a survey of 49 NYC EDs, 19 have changed coding practices or Health Information Systems since 2008. CPA identified a coincident and significant shift in word count for 8 of these hospitals. CPA also detected significant shifts in word count for 13 hospitals that did not report any changes. Figure 1 shows the results of CPA from one ED in NYC We observed immediate changes in daily syndrome count after the detected change in CC word count. For example, respiratory syndrome count increased with increased word count and decreased with decreased word count for 10 of the 21 EDs with a significant change in word count. Only 2 EDs saw an opposite effect on respiratory syndrome count. Meanwhile, 9 EDs saw no obvious change in respiratory syndrome count. Furthermore, these changes in daily CC word count coincided with subsequent changes in syndrome composition, the breakdown of syndromes into constituent symptoms. CONCLUSIONS: Change Point Analysis may be a useful method for prospectively detecting shifts in CC word count, which might represent a change in ED practices. In some instances changes to CC word count had an apparent effect on both syndrome capture and syndrome composition. Further studies are required to determine how often these changes happen and how they may affect the quality of syndromic surveillance.