Variation in the Assessment of Immune-Related Adverse Event Occurrence, Grade, and Timing in Patients Receiving Immune Checkpoint Inhibitors

IMPORTANCE: Toxic effects of conventional chemotherapy and molecularly targeted cancer therapies are generally well defined and occur at predictable points. By contrast, owing to their heterogeneous manifestations, unpredictable timing, and clinical overlap with other conditions, immune-related adverse events (irAE) may be more difficult to diagnose and characterize. OBJECTIVE: To determine concordance of algorithm-driven medical record review by medical oncologists for the characterization of 8 irAE in patients treated with immune checkpoint inhibitors. DESIGN, SETTING, AND PARTICIPANTS: Cross-sectional study of patients treated with immune checkpoint inhibitors at a National Cancer Institute–designated comprehensive cancer center from November 30, 2015, to March 7, 2018. A sample size of 52 patients provided 80% power to distinguish substantial agreement (κ = 0.85) from poor agreement (κ = 0.5) based on the Cohen κ. MAIN OUTCOMES AND MEASURES: Interrater agreement of 2 observers in the occurrence and grade of irAE. RESULTS: Of 52 patients (32 [61.5%] male; mean [SD] age, 69 [9] years) analyzed, 42 (80.8%) had non–small cell lung cancer and all received anti–programmed cell death 1 or anti–programmed cell death ligand 1 antibodies, with 3 patients (5.8%) receiving combinations with anti–cytotoxic T-lymphocyte antigen 4 antibodies. A median (interquartile range) of 82 (47-180) documents were reviewed per case. There was limited or poor interrater agreement on irAE occurrence (Cohen κ, 0.37-0.64), with the exception of hypothyroidism (κ = 0.8). Weighted κ similarly showed limited or poor agreement for irAE grade (κ = 0.31-0.75). Differences in assessed time of onset ranged from 5 to 188 days. As a control for data availability and access, observers had a high degree of agreement for the exact start date (98%) and end date (96%) of immunotherapy administration, suggesting that information interpretation rather than identification largely accounted for assessment differences. In multivariable analysis, therapy duration (adjusted odds ratio, 4.80; 95% CI, 1.34-17.17; P = .02) and Charlson Comorbidity Index (adjusted odds ratio, 4.09; 95% CI, 1.10-15.18; P = .03) were significantly associated with discordant irAE assessment. CONCLUSIONS AND RELEVANCE: These findings underscore critical challenges in assessing the occurrence, type, timing, and severity of irAE. Apart from hypothyroidism (a condition that has a discrete diagnostic laboratory test and few other likely etiologies during immunotherapy treatment), interobserver agreement was poor. Given the importance of accurate and timely assessment of toxic effects for clinical trials and real-world disease management, efforts to improve irAE diagnosis and characterization are needed.