Application of the Multi-Facet Rasch Model to Evaluate the Chief Resident Selection Survey: A Two-Year Study

Introduction The Chief Resident (CR) selection process is described by many residency programs as a collective effort from the residency program leadership, key faculty members, and resident peers. Unfortunately, the literature does not show any established guidelines, methods, or psychometric sound instruments to aid this process. The purpose of this study was to evaluate the properties of the newly developed CRs selection survey across two years using the Multi-Facet Rasch Model (MFRM). Methods This study used the MFRM to analyze two-year data from the newly developed CRs selection survey. After the first implementation of the tool in 2015, this instrument had its second-round evaluation process for the CRs selection in 2016. We applied a three-facet Rasch model (candidates, questions, and raters). We used Facets v. 3.66 and SAS 9.4 (SAS Institute Inc., Cary, NC) for data analysis. Results In 2015, 40 out of100 residents completed the survey to select three of the four candidates for the 2017-2018 CRs positions. The mean rating for each candidate showed that Candidate 1 received the highest rating of 5.56 while Candidates 2 and 4 received the exact same ratings. The majority of survey items performed very well based on the results from the MFRM while leaving room for improvement for a few items. In 2016, 55 out of 100 residents completed the revised survey to select three of the six candidates for the 2018-2019 CR positions. The mean rating showed that Candidate 3 received the highest mean rating of 5.81 while Candidate 2 received the lowest mean rating of 5.12. The item reliability was improved from 0.70 to 0.88 based on the results from the revised survey. The results were used to help inform decisions regarding the selection of chief residents. Conclusions The CR selection process requires a fair and collective effort from program leadership, relevant faculty members, and input from the resident group. Our study demonstrated that the survey tool we developed is appropriate to select CR candidates and MFRM is a promising technique in survey development and the evaluation of survey items.