Outcome-Based Validity and Reliability Assessment of Raters Regarding the Admission Triage Level in the Emergency Department: a Cross-Sectional Study

INTRODUCTION: Emergency department (ED) is usually the first line of healthcare supply to patients in non-urgent to critical situations and, if necessary, provides hospital admission. A dynamic system to evaluate patients and allocate priorities is necessary. Such a structure that facilitates patients’ flow in the ED is termed triage. OBJECTIVE: This study was conducted to investigate the validity and reliability of implementation of Emergency Severity Index (ESI) system version 4 by triage nurses in an overcrowded referral hospital with more than 80000 patient admissions per year and an average emergency department occupancy rate of more than 80%. METHOD: This prospective cross-sectional study was conducted in a tertiary care teaching hospital and trauma center with an emergency medicine residency program. Seven participating expert nurses were asked to assess the ESI level of patients in 30 written scenarios twice within a three-week interval to evaluate the inter-rater and intra-rater reliability. Patients were randomly selected to participate in the study, and the triage level assigned by the nurses was compared with that by the emergency physicians. Finally, based on the patients’ charts, an expert panel evaluated the validity of the triage level. RESULTS: During the study period, 527 patients with mean age of 54 ± 7 years, including 253 (48%) women and 274 (52%) men, were assessed by seven trained triage nurses. The degree of retrograde agreement between the collaborated expert panel’s evaluation and the actual triage scales by the nurses and physicians for all 5 levels was excellent, with the Cohen’s weighted kappa being 0.966 (CI 0.985–0.946, p < 0.001) and 0.813 (CI 0.856–0.769, p<0.001), respectively. The intra-rater reliability was 0.94 (p < 0.0001), and the inter-rater reliability for all the nurses was in perfect agreement with the test result (Cohen’s weighted kappa were as follows: 0.919, 0.956, 0.911, 0.955, 0.860, 0.956, and 0.868; p < 0.001). CONCLUSION: The study findings showed that there was perfect reliability and, overall, almost perfect validity for the triage performed by the studied nurses.