Implementation of the Radiological Society of North America Expert Consensus Guidelines on Reporting Chest CT Findings Related to COVID-19: A Multireader Performance Study

BACKGROUND: RSNA expert consensus guidelines provide a framework for reporting CT findings related to COVID-19, but have had limited multireader validation. PURPOSE: To assess the performance of the RSNA guidelines and quantify interobserver variability in application of the guidelines in patients undergoing chest CT for suspected COVID-19 pneumonia. MATERIALS AND METHODS: A retrospective search from 1/15/20 to 3/30/20 identified 89 consecutive CT scans whose radiological report mentioned COVID-19. One positive or two negative RT-PCR tests for COVID-19 were considered the gold standard for diagnosis. Each chest CT scan was evaluated using RSNA guidelines by 9 readers (6 fellowship trained thoracic radiologists and 3 radiology resident trainees). Clinical information was obtained from the electronic medical record. RESULTS: There was strong concordance of findings between radiology training levels with agreement ranging from 60 to 86% among attendings and trainees (kappa 0.43 to 0.86). Sensitivity and specificity of “typical” CT findings for COVID-19 per the RSNA guidelines were on average 86% (range 72%-94%) and 80.2% (range 75-93%), respectively. Combined “typical” and “indeterminate” findings had a sensitivity of 97.5% (range 94-100%) and specificity of 54.7% (range 37-62%). A total of 163 disagreements were seen out of 801 observations (79.6% total agreement). Uncertainty in classification primarily derived from difficulty in ascertaining peripheral distribution, multiple dominant disease processes, or minimal disease. CONCLUSION: The “typical appearance” category for COVID-19 CT reporting has an average sensitivity of 86% and specificity rate of 80%. There is reasonable interreader agreement and good reproducibility across various levels of experience.