Verifying the value of existing frameworks for formulary review at a large academic health system: assessing inter-rater reliability

BACKGROUND: The value assessment framework (VAF) is one approach to assessing the evidence and value of medications. VAFs are a way to measure and communicate the value of medications and other health care technologies for decision-making purposes. Given the increasing number of high-cost medications, challenging formulary inquiries, and critiques of currently available tools, health systems need to explore a standardized way to incorporate value assessment into formulary decision making. OBJECTIVES: To (a) evaluate existing VAFs by measuring inter-rater reliability among typical clinicians completing formulary reviews and (b) explore general implications of applying these tools to formulary decision making for all medications at a large academic health system. METHODS: This was a retrospective, observational study at a single health system. A list of medications added, denied, and removed from the system formulary from September 1, 2013, through August 31, 2018, was collected. Published VAFs, such as the American Society of Clinical Oncology (ASCO) Value Framework, European Society of Medical Oncology (ESMO) Magnitude of Clinical Benefit Scale, National Comprehensive Cancer Network (NCCN) Evidence Blocks, American College of Cardiology/American Heart Association Value Framework, and the incremental cost-effectiveness ratio (ICER) calculation were applied by 3 different reviewer groups. The primary outcome was inter-rater reliability among the 3 different reviewers for a given framework. Cohen’s weighted kappa and the intraclass correlation coefficient (ICC) were used to assess inter-rater reliability. RESULTS: The frameworks were applied to 94 medications. The VAFs with the highest ICCs between all 3 raters were NCCN (0.635; 95% CI = 0.387-0.823) and ASCO (0.634; 95% CI = 0.370-0.832), both indicating moderate inter-rater reliability. The VAFs with the lowest ICCs were ESMO (0.368; 95% CI = 0.126-0.611) and ICER (0.159; 95% = CI −0.018-0.365), with ICCs corresponding to poor reliability. CONCLUSIONS: Because high-cost medications are a challenge to health systems, VAFs may be beneficial to target formulary decision making in this setting. Applying VAFs proactively may improve interrater reliability and usability in formulary decision making.