The Effect of Changing Fellowship Interview Format on Candidate Ranking Variabilities: The COVID-19 Experience

PURPOSE: This study investigated the effect of interview format changes (in-person to virtual, one-to-one to multiple-to-one) necessitated by the COVID-19 travel restrictions on preliminary fellowship candidate ranking variabilities. DESIGN: Cross-sectional observational study. METHOD: In 2018 and 2019, the glaucoma fellowship interviews were conducted in-person in a one-to-one format, whereas in 2020, interviews were virtual and in a multiple (interviewers)-to-one (candidate) format. We compared ranking ranges of interviewers within the same virtual room (WSR) and not within the same virtual room (NWSR) to assess the effect of WSR versus NWSR on ranking variabilities. We also compared ranking categories (“accept,” “alternate,” and “pass”) agreements between 2018, 2019, and 2020 to assess the effect of virtual versus in-person interviews on ranking variabilities. RESULTS: NWSR and WSR mean rankings differed by 1.33 (95% confidence interval difference 0.61 to 2.04, p = 0.0003), with WSR interviewers having less variability than NWSR pairs. The variability between 2018/2019 (in-person interviews) and 2020 (virtual interviews) showed no differences between in-person and virtual interviews (weighted Kappa statistic 0.086 for 2018, 0.158 for 2019, and 0.101 for 2020; p < 0.05 for all years). The overall least attractive candidate has the lowest variability; the most attractive candidate has the second lowest variability. CONCLUSION: Grouping interviewers WSR during the interview decreased ranking variabilities compared to NWSR, while a change from in-person to virtual interview format did not increase the ranking variabilities. This suggests that the decreased nonverbal interactions in virtual interviews do not decrease interviewers' perceptions as applied to preliminary rankings.