Development and psychometric validation of a novel patient survey to assess perceived quality of substance abuse treatment in South Africa

BACKGROUND: A hybrid performance measurement system that combines patient-reported outcome data with administrative data has been developed for South African substance abuse treatment services. This paper describes the development and psychometric validation of one component of this system, the South African Addiction Treatment Services Assessment (SAATSA). METHODS: First, a national steering committee identified five domains and corresponding indicators on which treatment quality should be assessed. A decision was made to develop a patient survey to assess several of these indicators. A stakeholder work group sourced survey items and generated additional items where appropriate. The feasibility and face validity of these items were examined during cognitive response testing with 16 patients. This led to the elimination of several items. Next, we conducted an initial psychometric validation of the SAATSA with 364 patients from residential and outpatient services. Exploratory (EFA) and confirmatory factor analyses (CFA) were conducted to assess the latent structure of the SAATSA. Findings highlighted areas where the SAATSA required revision. Following revision, we conducted another psychometric validation with an additional sample of 285 patients. We used EFA and CFA to assess construct validity and we assessed reliability using Cronbach’s measure of internal consistency. RESULTS: The final version of the SAATSA comprised 31 items (rated on a four-point response scale) that correspond to six scales. Four of these scales are patient-reported outcome measures (substance use, quality of life, social connectedness and HIV risk outcomes) that together assess the perceived effectiveness of treatment. The remaining two scales assess patients’ perceptions of access to and quality of care. The models for the final revised scales had good fit and the internal reliability of these scales was good to excellent, with Cronbach’s α ranging from 0.72 to 0.89. CONCLUSION: A lack of adequate measurement tools hampers efforts to improve the quality of substance abuse treatment. Our preliminary evidence suggests that the SAATSA, a novel patient survey that assesses patients’ perceptions of the outcomes and quality of substance abuse treatment, is a psychometrically robust tool that can help fill this void.