Multi-Specialty Expert Physician Identification of Extranodal Extension in Computed Tomography Scans of Oropharyngeal Cancer Patients: Prospective Blinded Human Inter-Observer Performance Evaluation

BACKGROUND: Extranodal extension (ENE) is an important adverse prognostic factor in oropharyngeal cancer (OPC) and is often employed in therapeutic decision making. Clinician-based determination of ENE from radiological imaging is a difficult task with high inter-observer variability. However, the role of clinical specialty on the determination of ENE has been unexplored. METHODS: Pre-therapy computed tomography (CT) images for 24 human papillomavirus-positive (HPV+) OPC patients were selected for the analysis; 6 scans were randomly chosen to be duplicated, resulting in a total of 30 scans of which 21 had pathologically-confirmed ENE. 34 expert clinician annotators, comprised of 11 radiologists, 12 surgeons, and 11 radiation oncologists separately evaluated the 30 CT scans for ENE and noted the presence or absence of specific radiographic criteria and confidence in their prediction. Discriminative performance was measured using accuracy, sensitivity, specificity, area under the receiver operating characteristic curve (AUC), and Brier score for each physician. Statistical comparisons of discriminative performance were calculated using Mann Whitney U tests. Significant radiographic factors in correct discrimination of ENE status were determined through a logistic regression analysis. Interobserver agreement was measured using Fleiss’ kappa. RESULTS: The median accuracy for ENE discrimination across all specialties was 0.57. There were significant differences between radiologists and surgeons for Brier score (0.33 vs. 0.26), radiation oncologists and surgeons for sensitivity (0.48 vs. 0.69), and radiation oncologists and radiologists/surgeons for specificity (0.89 vs. 0.56). There were no significant differences between specialties for accuracy or AUC. Indistinct capsular contour, nodal necrosis, and nodal matting were significant factors in regression analysis. Fleiss’ kappa was less than 0.6 for all the radiographic criteria, regardless of specialty. CONCLUSIONS: Detection of ENE in HPV+OPC patients on CT imaging remains a difficult task with high variability, regardless of clinician specialty. Although some differences do exist between the specialists, they are often minimal. Further research in automated analysis of ENE from radiographic images is likely needed.