The Reliability and Relevance of a Quality of Decision Making Instrument, Quality of Decision-Making Orientation Scheme (QoDoS), for Use During the Lifecycle of Medicines

Introduction: The Quality of Decision-Making Orientation Scheme (QoDoS) was developed to provide organisations involved in submission, approval and reimbursement of new medicines with a tool to improve the quality of their decision-making processes and is considered the most promising tool for such purpose. This study aimed to further establish the measurement properties of the QoDoS by evaluating its reliability (internal consistency and test-retest reliability) and relevance in the target population. Methods: The study participants consisted of 55 individuals recruited from pharmaceutical companies, regulatory and HTA agencies. It was designed as a longitudinal study with participants assessed on two different occasions, at baseline (test 1) and then 7 days later (test 2). Internal consistency reliability was assessed with Cronbach’s alpha and the test-retest reliability was evaluated using the intraclass correlation coefficients (ICC) based on absolute agreement, 2 way mixed-effects model for the four QoDoS domains. The relevance of the QoDoS was evaluated by applying cognitive debriefing using five short feedback questions following test 1. Results: Test 1 was completed by 44 study participants (80% response rate) and test 2 was completed by 32 of the 44 individuals, resulting in a 73% response rate. Cronbach’s alpha coefficient was greater than 0.7 across all the domains for test 1 and test 2, ranging from 0.71 to 0.79, indicating good consistency of responses. For the overall score across all 47 items, the Cronbach’s alpha coefficient was 0.81 for test 1 and 0.86 for test 2, which is rated as very good. The four QoDoS domains showed moderate to strong reproducibility (ICC range: 0.63–0.86). The outcome of the cognitive debriefing from the 43 respondents (98% response rate) confirmed the relevance (95% agreement), language clarity (95%) and completeness of items (86%); the clarity of the scaling (91%) as well as spontaneity of responses (95%). Conclusion: These results provide strong support for the relevance and reliability of the QoDoS, which are key properties for future longitudinal and cross-sectional applications of the instrument when evaluating quality of decision making by those involved in the lifecycle of medicines.