Current Milestones Assessment Practices, Needs, and Challenges of Program Directors: A Collective Case Study in a Pediatric Hospital Setting

Introduction Accreditation Council for Graduate Medical Education's (ACGME's) Milestones assessment requirement has placed new demands on Program Directors (PDs), especially those with limited knowledge of assessment and evaluation activities. There is a lack of clarity on how Program Director (PDs)/Associate PDs (APDs) are effectively implementing milestones assessment and evaluation practices in the Graduate Medical Education programs. The purpose of this study was to investigate current assessment practices, needs, and challenges of PDs in implementing milestones assessment within their residency and fellowship programs in a pediatric hospital setting. Methods This study used a collective case study approach to obtain information from PDs, APDs, and Clinical Competency Committee (CCC) Chairs in 19 graduate programs at a pediatric hospital. We used structured meetings with planned agendas and a pre-formatted template to itemize program needs/difficulties/challenges in the milestone assessment. We used cross-case thematic content anal­ysis to identify categories and themes to compare differences and commonalities across programs. Results A total of 38 PDs, APDs, and CCC Chairs from 19 different specialties/subspe­cialties participated in this study. Thirteen types of assessment and evaluation tools were consistently used across programs. Three categories emerged in relation to those assessment and evaluation types (direct, indirect, and multi-source). Rotation evaluation (84.2%), direct observation (73.2%), and 360-degree assessment (68.4%) were primarily used for measuring patient care among the six core competencies. Programs’ needs varied from curriculum and assessment tool development to alignment of milestones items, and to creating a sys­tematic assessment management plan. The most common challenges were difficulties related to logistics and tracking of evaluation in the survey management system (52.6%), challenges with time management (47.3%), and difficulty in determining and interpret­ing the milestones’ numbers and levels (31.5%). Conclusions Milestones assessment and evaluation in medical education can be a challenge, but a priority for many training programs. Our study indicated that milestones assessment and evaluation in medical education are far more com­plex than we expect. Multiple assessment methods must be utilized to evaluate all essential competencies for accurate measurement of trainees’ performance abilities. Our study uncovered several issues PDs faced during the implementation of milestones assessment and needs and challenges.