When (Almost) Everyone is Above Average: A Critical Analysis of American Orthopaedic Association Committee of Residency Directors Standardized Letters of Recommendation

The American Orthopaedic Association introduced standardized letters of recommendations (SLORs) to improve on traditional letters of recommendations by “providing a global prospective on an applicant.” However, no study has defined the utilization of SLORs, the distribution of applicant ratings in SLORs, or the impact of sex, race, or degree of involvement between the letter writer and applicant on SLOR domain ratings. METHODS: One-hundred seventy-nine applications were randomly selected from all applicants submitted to a single, academic orthopaedic residency program. A single reviewer extracted both applicant characteristics and SLOR characteristics from applications. Descriptive statistics, chi-square tests, and nonparametric one-way analysis of variance analysis were conducted. RESULTS: Six hundred twenty-eight letters of recommendation from 179 applicants were analyzed. Four hundred ninety-seven of 628 (79.1%) letters contained a SLOR. Mean percentile ratings were calculated for all the following domains: patient care (mean ± SD = 86.7 ± 8.7), medical knowledge (87.2 ± 8.6), interpersonal and communication (87.7 ± 9.3), procedural (86.6 ± 8.9), research (88.9 ± 9.0), ability to work within a team (89.6 ± 8.4), professionalism (90.8 ± 7.3), initiative and drive (90.6 ± 7.6), and commitment to orthopaedic surgery (91.1 ± 6.7). Forty-eight percent of applicants were indicated as “ranked to guarantee match.” When compared with male applicants, female applicants demonstrated higher percentile ratings in patient care (88.6 ± 8.2 vs. 86.3 ± 8.7, p = 0.010), interpersonal and communication skills (90.6 ± 7.3 vs. 86.9 ± 9.6, p < 0.001), and ability to work within a team (91.3 ± 6.3 vs. 89.2 ± 8.8, p = 0.045). Higher United States Medical Licensing Examination step 1 (r = 0.08, p = 0.05) and step 2 scores (r = 0.10, p = 0.02) correlated with higher medical knowledge ratings. The number of publications (r = 0.3, p < 0.001) and presentations (r = 0.25, p < 0.001) correlated with research ratings. CONCLUSION: SLORs demonstrated a profound ceiling effect, potentially limiting the utility of the instrument for the comparison of applicants. Future modifications to this instrument may include measures to better delineate between applicants.