Reference letters for subspecialty medicine residency positions: are they valuable for decision-making? Results from a Canadian study

BACKGROUND: The letter of recommendation is currently an integral part of applicant selection for residency programs. Internal medicine residents will spend much time and expense completing sub-specialty away electives to obtain a letter of recommendation. The purpose of this study was 1) to examine a large sample of reference letters in order to define essential components of a high-quality letter, and 2) to elucidate the relationship between quality of reference letter and the letter writer. METHODS: We conducted a two-phase study. In phase one, a large sample of letters of recommendation was examined using an audit tool as a coding framework. A 5-point composite endpoint of high-quality letter components was subsequently developed. In phase two, program director letters were compared to non-program director home institution and non-home institution elective letters based on inclusion of components of the 5-point composite endpoint using Chi square testing. RESULTS: 715 letters were examined (398 non-program director home institution letters, 201 program director letters, and 116 non-home institution elective letters). High-quality letter components were: nature of relationship, duration of relationship, In Training Evaluation Report information, research involvement and comments on areas for improvement. Program director letters had a significantly higher proportion (10.4%) of all 5 high-quality components, compared to 0% in both non-program director home institution letters and elective letters (p < 0.001). A significantly higher proportion of program director letters had 4–5 high-quality components (62.5%) compared to 2% of non-program director home institution letters and 0% of elective letters (p < 0.0001). CONCLUSIONS: Letters of recommendation from elective rotations are of the poorest quality and such rotations should not be pursued for the sole purpose of obtaining a letter. The low quality of elective letters leads to the recommendation that writers should decline to write them, programs should not require them and trainees should not request them. Program directors write the highest quality letters and should be a resource for faculty development. Clinical supervisors can use the 5-point composite endpoint as a guide when writing letters for applicants.