Optimizing Reliability of Real-Time Sonographic Examination and Scoring of Joint Synovitis in Rheumatoid Arthritis

Objective: Musculoskeletal ultrasound real-time image acquisition and scoring are complex, and many factors affect reliability. Static image reliability does not guarantee real-time scoring. This study aimed to identify factors and solutions to improve real-time scoring reliability for the grey scale and power Doppler evaluation of synovitis. We also report on using a novel musculoskeletal ultrasound synovitis rule-based scoring atlas. Methods: In four stages, we evaluated inter- and intra-reader reliability among three ultrasonographers (US1-3). Intra- and inter-reader reliability was calculated using weighted-kappa, intraclass correlation coefficient, and Spearman correlation. Reliability statistics were compared between stages using permutation tests to compute empirical distributions for differences in those statistics. At each stage, factors that diminished reliability were identified and addressed. After intensive reliability exercises, a RA MSUS atlas with in-depth scoring rules was generated to improve interpretive reliability. Results: The three ultrasonographers had good to excellent intra-reader reliability for real-time acquisition scoring over 2432 views (weighted kappa 0.52-0.80, intraclass correlation coefficient 0.59-0.86, and Spearman correlation 0.64-0.86). Inter-reader reliability was good to excellent between US1/US2 and US1/US3 (weighted kappa 0.51-0.66, intraclass correlation coefficient 0.66-0.75, Spearman correlation 0.59-0.73). US1 achieved significant improvement in intra-reader reliability from stage 1 to stage 2 (p<0.05, weighted-kappa 0.63 to 0.80, intraclass correlation coefficient 0.71 to 0.86, Spearman 0.67 to 0.86) with use of the atlas. Conclusion: This rheumatoid arthritis musculoskeletal ultrasound study addressed complex factors affecting musculoskeletal ultrasound acquisition-scoring reliability. Systematic identification and amelioration of many factors and using a novel rule-based scoring atlas significantly improved intra-reader reliability.