Milestones-based direct observation tools in internal medicine resident continuity clinic

BACKGROUND: With milestones-based assessment, there is an increased need for tools to facilitate direct observation of clinical trainees. This study was designed to compare a Mini-CEX tool to new direct observation tools (DOTs) linked to internal medicine milestones. METHODS: A web based survey was used to examine satisfaction and usefulness of DOTs compared to the Mini-CEX. Residents and preceptors were surveyed three times over 6 months with half serving as control (using mini-CEX) compared to those using the DOTs. Likert scale quantitative answers and qualitative comments were analyzed using generalized estimating equations. RESULTS: Out of 94 residents and 32 faculty 81 and 90% completed the survey for at least one time point. In adjusted models, there was no significant change in resident evaluation comparing the tools on a number of questions including overall satisfaction and resident perception of receiving high quality feedback. By contrast, faculty preceptors reported increased ratings on many of the questions evaluating their use of the new tools including ability to provide higher quality feedback and overall satisfaction. A number of challenges and benefits of the new tools were identified in qualitative feedback by both residents and preceptors. CONCLUSIONS: All parties recognized the value and limitations of direct observation. Overall these new office based DOTs were perceived similarly to the mini-CEX by residents while faculty reported higher satisfaction. The DOTs are a useful addition to the tool box available for the assessment of clinical skills of medical trainees, especially from the viewpoint of faculty preceptors. ELECTRONIC SUPPLEMENTARY MATERIAL: The online version of this article (10.1186/s12909-017-1077-y) contains supplementary material, which is available to authorized users.