The neglect of treatment-construct validity in psychotherapy research: a systematic review of comparative RCTs of psychotherapy for Borderline Personality Disorder

BACKGROUND: Randomized controlled trials (RCTs) are considered the best methodology for studying the efficacy of psychotherapy. Optimally an RCT design makes it possible to conclude that if one treatment has a better outcome than another, this is due to the treatment package (TP) as it was implemented in this particular context, rather than other factors beyond the treatment (= high internal validity). Strong internal validity does not, however, provide evidence for the treatment model (TM) that provides the theoretical basis of the TP, because the TP that is tested may differ from the comparison condition in a number of other ways that suggest alternative explanations for the effects. These alternative treatment contrasts represent threats to construct validity of the conclusions. Maximal construct validity requires (1) that the treatments are clearly contrasted on the experimental factors (treatment integrity), and (2) that alternative treatment contrasts can be eliminated. The analysis of alternative explanations is a neglected topic in psychotherapy research. To approach this problem, a methodology for the analysis of treatment contrasts is suggested and tested. METHODS: Two indexes were defined: (1) a Treatment Integrity Index (TII) and (2) an Alternative Treatment Contrast Index (ATCI). This methodological approach was applied to eight comparative RCTs of treatments for Borderline Personality Disorder (BPD), which were coded for a set of treatment contrasts independently by three coders. RESULTS: The analysis of the RCTs of treatments for BPD showed that construct validity differed widely between the different studies but was generally low (low TII and ATCI), and that it is therefore difficult to draw causal conclusions from this research. The publication policies of scientific journals in this area seldom require the systematic data relevant to an analysis of alternative explanations of the effects, which is needed to provide evidence for a particular TM. CONCLUSIONS: Research on psychotherapy needs to be refocused from treatment packages (TP) to treatment models (TM). This requires an improved conceptualization of the methodological principles and skills involved, and the development of valid measures of these, but also improved reporting standards concerning treatment-construct validity in scientific journals. ELECTRONIC SUPPLEMENTARY MATERIAL: The online version of this article (doi:10.1186/s40359-016-0151-2) contains supplementary material, which is available to authorized users.