How can medical residency candidates be evaluated more reliably?

 Standardized tests as measures of physician aptitude are falling into disrepute and disuse.  Consequently the medical profession needs to develop better ways for evaluators (e.g. med school professors) to communicate information about applicants to residency programs.

Here are two reflections on the current state of afairs in Orthopaedic surgery.

Are Narrative Letters of Recommendation for Medical Students Interpreted as Intended by Orthopaedic Surgery Residency Programs?  by Egan, Cameron R. MD; Dashe, Jesse MD; Hussein, Amira I. PhD; Tornetta, Paul III MD

Clinical Orthopaedics and Related Research: February 25, 2021 - doi: 10.1097/CORR.0000000000001691

"Background: Narrative letters of recommendation are an important component of the residency application process. However, because narrative letters of recommendation are almost always positive, it is unclear whether those reviewing the letters understand the writer’s intended strength of support for a given applicant.

"Questions/purposes: (1) Is the perception of letter readers for narrative letters of recommendation consistent with the intention of the letter’s author? (2) Is there inter-reviewer consistency in selection committee members’ perceptions of the narrative letters of recommendation?

"Methods: Letter writers who wrote two or more narrative letters of recommendation for applicants to one university-based orthopaedic residency program for the 2014 to 2015 application cycle were sent a survey linked to a specific letter of recommendation they authored to assess the intended meaning regarding the strength of an applicant. A total of 247 unstructured letters of recommendation and accompanying surveys were sent to their authors, and 157 surveys were returned and form the basis of this study (response percentage 64%). The seven core members of the admissions committee (of 22 total reviewers) at a university-based residency program were sent a similar survey regarding their perception of the letter.

...

"Conclusion :Our results demonstrate that the reader’s perception of narrative letters of recommendation did not correlate well with the letter writer’s intended meaning and was not consistent between letter readers at a single university-based urban orthopaedic surgery residency program.

"Clinical Relevance: Given the low correlation between the intended strength of the letter writers and the perceived strength of those letters, we believe that other options such as a slider bar or agreed-upon wording as is used in many dean’s letters may be helpful."

**********

CORR Insights®: Are Narrative Letters of Recommendation for Medical Students Interpreted as Intended by Orthopaedic Surgery Residency Programs? by Zywiel, Michael G. MD, MSc, Clinical Orthopaedics and Related Research: April 29, 2021 - doi: 10.1097/CORR.0000000000001780

"With the upcoming transition of the USMLE Step 1 to a pass/fail score, and as we continue to gather more evidence calling into question the current selection criteria used for surgical training, programs are increasingly left to wonder how they can select learners that are most likely to succeed. Similarly, learners are increasingly left wondering how they can appropriately determine whether they are likely to succeed in a chosen specialty.

...

"Going forward, we need more research within the domain of selection criteria for training. This includes identifying more reliable predictors of technical skill, nontechnical skill, as well as performance in independent practice. The failure of most current selection criteria to adequately predict performance suggests that novel, specialty-specific instruments may need to be developed, evaluated, and ultimately incorporated at the medical student level to better predict future performance."