Agreement between neuroimages and reports for natural language processing-based detection of silent brain infarcts and white matter disease

BACKGROUND: There are numerous barriers to identifying patients with silent brain infarcts (SBIs) and white matter disease (WMD) in routine clinical care. A natural language processing (NLP) algorithm may identify patients from neuroimaging reports, but it is unclear if these reports contain reliable information on these findings. METHODS: Four radiology residents reviewed 1000 neuroimaging reports (RI) of patients age > 50 years without clinical histories of stroke, TIA, or dementia for the presence, acuity, and location of SBIs, and the presence and severity of WMD. Four neuroradiologists directly reviewed a subsample of 182 images (DR). An NLP algorithm was developed to identify findings in reports. We assessed interrater reliability for DR and RI, and agreement between these two and with NLP. RESULTS: For DR, interrater reliability was moderate for the presence of SBIs (k = 0.58, 95 % CI 0.46–0.69) and WMD (k = 0.49, 95 % CI 0.35–0.63), and moderate to substantial for characteristics of SBI and WMD. Agreement between DR and RI was substantial for the presence of SBIs and WMD, and fair to substantial for characteristics of SBIs and WMD. Agreement between NLP and DR was substantial for the presence of SBIs (k = 0.64, 95 % CI 0.53–0.76) and moderate (k = 0.52, 95 % CI 0.39–0.65) for the presence of WMD. CONCLUSIONS: Neuroimaging reports in routine care capture the presence of SBIs and WMD. An NLP can identify these findings (comparable to direct imaging review) and can likely be used for cohort identification. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1186/s12883-021-02221-9.