Assessment of agreement using the equine glandular gastric disease grading system in 84 cases

INTRODUCTION: Equine glandular gastric disease (EGGD) is a common condition causing signs of gastric pain although lesions are highly variable in their appearance. The only definitive method to diagnose EGGD ante‐mortem is gastroscopy. The current recommended method for describing these lesions is the European College of Equine Internal Medicine (ECEIM) guidelines; however, repeatability between users is variable. This study aimed to validate the reliability of lesion descriptions using ECEIM consensus guidelines, using four blinded equine internal medicine diplomates. METHODS: Ninety‐two horses with EGGD with pre‐ and post‐treatment gastroscopy images were identified using the electronic record at a UK equine hospital between 2012 and 2019. Eight horses were excluded due to non‐diagnostic images. Four blinded observers used the recommended grading system to describe images and outcomes. Intraclass correlation coefficients and Krippendorff's alpha were used to determine reliability and agreement, respectively. RESULTS: Intraclass correlation coefficient for severity was 0.782 (95% confidence interval [CI] 0.722–0.832), for distribution was 0.671 (95% CI 0.540–0.763), for the descriptor raised was 0.635 (95% CI 0.479–0.741), fibrinosuppurative was 0.745 (95% CI 0.651–0.812), haemorrhagic was 0.648 (95% CI 0.513–0.744), hyperaemic was 0.389 (95% CI 0.232–0.522) and for outcome was 0.677 (95% CI 0.559–0.770). Krippendorff's alpha for severity was 0.466 (95% CI 0.466–0.418), for distribution was 0.304 (95% CI 0.234–0.374), for the descriptor raised was 0.268 (95% CI 0.207–0.329), fibrinosuppurative was 0.406 (95% CI 0.347–0.463), haemorrhagic was 0.287 (95% CI 0.229–0.344), hyperaemic was 0.112 (95% CI 0.034–0.188) and for outcome was 0.315 (95% CI 0.218–0.408). There was moderate reliability determined between observers using intra‐class correlation coefficients and unacceptable agreement determined between observers using Krippendorff's alpha. DISCUSSION: These results suggest that the current grading system is not comparable between observers, indicating the need to review the grading system or define more robust criteria.