Quality of out-of-hours telephone triage by general practitioners and nurses: development and testing of the AQTT – an assessment tool measuring communication, patient safety and efficiency

Objective: To develop a valid and reliable assessment tool able to measure quality of communication, patient safety and efficiency in out-of-hours (OOH) telephone triage conducted by both general practitioners (GP) and nurses. Design: The Dutch KERNset tool was translated into Danish and supplemented with items from other existing tools. Face validity, content validity and applicability in OOH telephone triage (OOH-TT) were secured through a two-round Delphi process involving relevant stakeholders. Forty-eight OOH patient contacts were assessed by 24 assessors in test-retest and inter-rater designs. Setting: OOH-TT services in Denmark conducted by GPs, nurses or doctors with varying medical specialisation. Patients: Audio-recorded OOH patient contacts. Main outcome measures: Test-retest and inter-rater reliability were analysed using ICC(agreement), Fleiss’ kappa and percent agreement. Results: Major adaptations during the Delphi process were made. The 24-item assessment tool (Assessment of Quality in Telephone Triage – AQTT) measured communicative quality, health-related quality and four overall quality aspects. The test-retest ICC(agreement) reliability was good for the overall quality of communication (0.85), health-related quality (0.83), patient safety (0.81) and efficiency (0.77) and satisfactory when assessing specific aspects. Inter-rater reliability revealed reduced reliability in ICC(agreement) and in Fleiss’ kappa. Percent agreement revealed satisfactory agreements when differentiating between ‘poor’ and ‘sufficient’ quality). Conclusion: KEYPOINTS: Comparative knowledge is sparse regarding quality of out-of-hours telephone triage conducted by general practitioners and nurses. The assessment tool (AQTT) enables assessment of quality in OOH telephone triage conducted by nurses and general practitioners. AQTT is feasible and clinically relevant for assessment of communication, patient safety and efficiency. AQTT can be used to identify areas for improvement in telephone triage.