Signaling in medical residency applications

 We're starting to see descriptive studies of how signals are being used in the labor market for new doctors.  Each medical specialty has chosen to adapt the kinds of signals used in Economics in its own way, with some specialties using only a handful of signals and others eliciting as many as 30.

Here are two papers from a recent issue of Academic Medicine.

Impact of Applicants’ Characteristics and Geographic Connections to Residency Programs on Preference Signaling Outcomes in the Match, by Benjamin, William J. MPH; Lenze, Nicholas R. MD, MPH; Bohm, Lauren A. MD; Thorne, Marc C. MD, MPH; Abraham, Reeni MD; Sepdham, Dan MD; Mihalic, Angela P. MD; Kupfer, Robbi A. MD,  Academic Medicine 99(4):p 437-444, April 2024. | DOI: 10.1097/ACM.0000000000005551

"Abstract

Purpose : To assess the impact of applicant and residency program characteristics on preference signaling outcomes in the Match during the first 2 years of implementation across 6 specialties.

Method : Data were obtained from the Texas Seeking Transparency in Application to Residency survey for applicants applying into otolaryngology during the 2020–2021 and 2021–2022 application cycles and into dermatology, internal medicine (categorical and preliminary year), general surgery, and urology during the 2021–2022 application cycle. The primary outcome was signal yield, defined as the number of interviews at signaled programs divided by the total number of signals sent. Associations with applicant-reported characteristics and geographic connections to residency programs were assessed using Wilcoxon rank sum testing, Spearman’s rank correlation testing, and ordinary least squares regression.

Results : 1,749 applicants with preference signaling data were included from internal medicine (n = 884), general surgery (n = 291), otolaryngology (n = 217), dermatology (n = 147), urology (n = 124), and internal medicine preliminary year (n = 86). On average 60.9% (standard deviation 32.3%) of signals resulted in an interview (signal yield). There was a stepwise increase in signal yield with the percentage of signals sent to programs with a geographic connection (57.3% for no signals vs. 68.9% for 5 signals, P < .01). Signal yield was positively associated with applicant characteristics, such as United States Medical Licensing Exam Step 1 and 2 scores, honors society membership, and number of publications (P < .01). Applicants reporting a lower class rank quartile were significantly more likely to have a higher percentage of their interviews come from signaled programs (P < .01).

Conclusions: Signal yield is significantly associated with geographic connections to residency programs and applicant competitiveness based on traditional metrics. These findings can inform applicants, programs, and specialties as preference signaling grows."

And here are the introductory paragraphs:

"The rising number of residency applications submitted per applicant has led to concerns that programs will not be able to adequately perform a holistic review of all applications and will instead rely on easily reviewed metrics, such as United States Medical Licensing Exam (USMLE) scores, class rank, and medical school reputation.1,2 In addition, COVID-19–related changes to the residency application process, such as the introduction of virtual interviewing and a cap on the number of away rotations medical students can complete, have limited applicants’ ability to informally express their interest in programs.3 Further, there is evidence that, while applying to the maximum number of programs is advantageous at the individual level, it leads to poorer overall results when all applicants follow this practice.2 To address this issue, new systems have been proposed, including personalized application paragraphs,4 program-specific messages,5 and preference signals.4,6,7

"Otolaryngology implemented a preference signaling system in 2021, which was based on theory developed by the American Economic Association (AEA) in 2006. The AEA used a preference signaling system for job market applicants, whereby applicants were allowed to express special interest in particular employers in their applications.8 Results from the AEA program highlighted that preference signals were beneficial to both candidates and employers in a labor market where employers are unable to provide full attention to every application they receive.9 Building off previous economic work, a computer simulation study run on 2014 otolaryngology Match data found that the number of interview invitations improved when applicants provided preferences on their Electronic Residency Application Service application; this result would have benefitted both programs and applicants.2

"Based on this research, the Otolaryngology Program Directors Association formally implemented a preference signaling system during the 2020–2021 application cycle in which applicants were granted 5 “signals” to send to residency programs prior to interviews indicating their strong interest in that program. Each program then received a list of the applicants who had sent them a signal.6,10,11 Data from the 2021 otolaryngology Match were notable for significantly increased interview rates at signaled programs across all levels of applicant competitiveness.10,12 Furthermore, the majority of program directors and applicants strongly supported the continuation of preference signaling.10,11 During the 2021–2022 application cycle, preference signaling pilot programs were implemented in 5 additional specialties: dermatology, internal medicine (categorical), internal medicine preliminary year, surgery (categorical), and urology, with each specialty using 5 signals per applicant, except dermatology, which used 3 signals"

#########

The Relationship Between Program and Applicant Characteristics With Applicant Program Signals in the 2022 Residency Recruitment Cycle: Findings From 3 Specialties, by LaFemina, Jennifer MD; Rosman, Ilana S. MD; Wallach, Sara L. MD; Wise, Paul E. MD; Smink, Douglas S. MD, MPH; Fletcher, Laura PhD, Academic Medicine 99(4):p 430-436, April 2024. | DOI: 10.1097/ACM.0000000000005586

"Abstract

Purpose: Continuing increases in application volume have driven a national dialogue to reform the residency recruitment process. Program signaling allows applicants to express interest in a program at the preinterview stage with the goal of helping programs identify applicants with more genuine interest in their programs. This study explored the relationship between program signals and program and applicant characteristics.

Method: Participating dermatology, general surgery, and categorical internal medicine (IM) programs and applicants of the 2022 supplemental ERAS application (SuppApp) were included. Data from the SuppApp, the MyERAS Application for Residency Applicants (MyERAS), and the 2020 GME Track Survey were used. Cohen’s h was used to determine effect size, and chi-squared was used to determine statistical significance.

Results:There was an uneven distribution of signals to programs, with 25% of programs receiving about half of the signals across all 3 specialties. Programs with larger numbers of both residents and applicants received greater numbers of program signals relative to their program density, although this effect was small (h < 0.50, P < .001). No meaningful differences were seen across genders for any specialty. Only Hispanic applicants in IM sent a higher proportion of signals to programs with more underrepresented in medicine residents than White only applicants (40% vs 26%, h = 0.30, P < .001). Across all specialties, there was a small-to-moderate effect for international medical graduate (IMG) applicants sending a larger proportion of signals to programs with more IMG residents (h < 0.80, P < .001).

Conclusions: This first-year pilot study (i.e., SuppApp) provided initial evidence that supports the feasibility and fairness of program signals in residency selection. As program signals become more common across specialties, future research should continue to evaluate trends in where applicants send signals, and possible relationships between program and application characteristics."

"IMG applicants were more likely to signal programs with a greater proportion of IMG residents. The effect was small in dermatology and increased to moderate in GS and large in IM. In the NRMP’s 2022 Main Residency Match, 11 IMGs (U.S. and non-U.S.) matched into postgraduate year 2 dermatology, representing 2% of positions. This compares to the 10% and 38% IMG Match rate into GS and IM, respectively.21 While at this time, correlation of signal distribution and the likelihood of successfully matching is not available, these findings suggest that in general, IMG applicants sent more signals to programs they knew to be “IMG friendly” (i.e., more likely to accept IMGs), which they could easily identify with tools such as the Residency Explorer Tool22 and the Residency Programs List.23 However, if IMGs continue to send more signals to programs with already higher proportions of IMG residents, this may maintain the status quo or even further restrict the IMG applicant pool all programs are willing to consider during their resident selection process because programs with fewer IMGs will continue to receive a lower proportion of signals from IMG applicants. This could ultimately negatively affect diversity across programs"