Evaluation of the reliability of the criteria for assessing prescription quality in Chinese hospitals among pharmacists in China

BACKGROUND: The reliability of Criteria for Assessing Prescription Quality in Chinese Hospitals (CAPQCH) has never been rigorously verified. This study was designed to verify the reliability of the CAPQCH among pharmacists in China. METHODS: Fourteen pharmacists, 5 from hospitals and 9 from the communities were recruited. We randomly selected 200 prescriptions, and made the testing prescriptions including appropriate and inappropriate testing prescriptions. Pharmacists assessed these testing prescriptions according to criteria in CAPQCH. Three test sets (Set 1, Set 2, and Set 3) were evaluated at 6-month intervals. Before administration of Set 3, pharmacists were informed that achievement on Set 3 would be reflected in their performance appraisal. We also evaluated the performance based on prescription comments before and after combining several confusing criteria. Cohen’s Kappa statistic, Fleiss’ Kappa statistic, and accuracy were employed to evaluate reliability among pharmacists. RESULTS: Median values of Cohen’s Kappa were 0.61 in Set 1, 0.66 in Set 2, and 0.80 in Set 3; reliability is thus substantial. Our data indicate no significant differences between Set 1 and Set 2, whereas Set 3 indicates significantly improved performance. Moreover, combinations of confusing criteria contributed little to improvement of performance in prescription comments. CONCLUSION: Our results verified the reliability of CAPQCH application by working pharmacists. Adding performance based on prescription comments to personal appraisals was effective in improving the quality of prescription comments. These findings may be useful when future modification of the CAPQCH is considered. Moreover, this study contributes to improving the understanding of the prescription assessment situation in China. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1186/s12913-022-07854-1.