Inter-Rater Agreement in the Assessment of Video Recordings of Eye Drop Instillation by Glaucoma Patients

PURPOSE: To create a standardized method for evaluating the video recordings of patients self-instilling eye drops and to determine the level of agreement of eye drop instillation efficacy, safety and efficiency ratings by three masked graders. DESIGN: Prospective cross-sectional study. PARTICIPANTS: 78 patients with open-angle glaucoma or ocular hypertension who had at least 6 months of experience with the use of eye drop medications. METHODS: Participants were video recorded while self-instilling artificial tears sequentially to both eyes. Three masked observers graded these video recordings on three criteria: efficacy (the determination of whether an eye drop was instilled on the ocular surface), safety (assessment of whether the tip of the medication bottle made contact with the ocular surface or eyelids), and efficiency (the number of eye drops expressed from the bottle). MAIN OUTCOME MEASURES: After grading the video recordings based on efficacy, safety, and efficiency, kappa statistics were used to estimate inter-rater agreement. RESULTS: The mean kappa level of agreement for efficacy, safety, and efficiency was 0.64 (95% confidence interval (CI), 0.42–0.87), 0.73 (95% CI, 0.58–0.88), and 0.62 (95% CI, 0.42–0.81), respectively. CONCLUSIONS: We demonstrated good inter-rater reproducibility of the masked analysis of video recordings of patients self-instilling eye drops based on three criteria: efficiency, safety, and efficacy.