Assessing doctors’ competencies using multisource feedback: validating a Japanese version of the Sheffield Peer Review Assessment Tool (SPRAT)

OBJECTIVE: To assess the validity and reliability of the Sheffield Peer Review Assessment Tool (SPRAT) Japanese version for evaluating doctors’ competencies using multisource feedback. METHODS: SPRAT, originally developed in the UK, was translated and validated in three phases: (1) an existing Japanese version of SPRAT was back-translated into English; (2) two expert panel meetings were held to develop and assure content validity in a Japanese setting; (3) the newly devised Japanese SPRAT instrument was tested by a multisource feedback survey, validity was tested using principal component factor analysis, and reliability was assessed using generalisability and decision studies based on generalisability theory. RESULTS: 86 doctors who had been practising for between 2 and 33 years participated as assessees and were evaluated with the SPRAT tool. First, the doctors identified 1019 potential assessors who were each sent SPRAT forms (response rate, 81%). The mean number of assessors per doctor was 9.7 (SD=2.5). The decision study showed that 95% CIs of ±0.5 were achieved with only 5 assessors. 85 of the 86 doctors achieved scores that could be placed with 95% CI above the 4 expected standard. Doctors received lower scores from more senior assessors (p<0.001) and higher scores from those they had known longer (p<0.001). Scores also varied with the job role (p<0.05). CONCLUSIONS: Following translation and content validation, the Japanese instrument behaved similarly to the UK tool. Assessor selection remains a primary concern, as the assessment scores are affected by the seniority of the assessor, the length of the assessor–assessee working relationship, and the assessor's job role. Users of the SPRAT tool need to be aware of these limitations when administering the instrument.