The bumpy road to achieve reliability of clinical profile characteristics in psychosis and related disorders

OBJECTIVES: Profile characteristics are factors that are relevant for diagnosis, prognosis or treatment. The present study aims to develop a set of clinically relevant profile characteristics. Moreover, our goal is to determine the inter‐rater reliability (IRR) of the selected profile characteristics. METHODS: Potential profile characteristics were determined by literature review. Assessment of IRR was done by comparing scores on profile characteristics determined by two researchers. We conducted three subsequent studies: (1) assessment of pre‐training IRR, (2) IRR following implementation of an instruction manual, (3) IRR after optimizing scoring methods. IRR was measured with the Intraclass Correlation Coefficient (ICC). RESULTS: IRR scores of profile characteristic Illegal activities were high across the three studies (ICC ≥ 0.75). Following training procedures in study 2 and 3, reliability estimates remained low to moderate (ICC < 0.75) for the profile characteristics Support of relatives, Aggression recent and lifetime, substance use and insight recent. IRR scores of the other eight profile characteristics varied from low, moderate to high across studies. CONCLUSION: IRR scores of profile characteristics were highly variable, and mostly inadequate in all three studies. Consequently, further research should focus on specification of severity scores of profile characteristics, optimizing scoring methods and re‐evaluation of IRR.