Many preference signals as a soft cap on number of applications in medical residency matching

 Here's a review article on matching for medical residents,  with particular attention to neurosurgery, in the Cureus Journal of Medical Science.  In specialties that (like neurosurgery) allow applicants to send many signals, many applicants signal to and match with programs with which they have some prior connection.

Ozair, Ahmad, Jacob T. Hanson, Donald K. Detchou, Matthew P. Blackwell, Abigail Jenkins, Marianne I. Tissot, Umaru Barrie et al. "Program Signaling and Geographic Preferences in the United States Residency Match for Neurosurgery." Cureus 16, no. 9 (2024).

Abstract: Postgraduate residency training has long been the cornerstone of academic medicine in the United States. The Electronic Residency Application Service (ERAS), managed by the Association of American Medical Colleges (AAMC), is the central residency application platform in the United States for most clinical specialties, with the National Residency Matching Program (NRMP) being the algorithm for matching residency programs with applicants. However, the determination of the best fit between ERAS applicants and programs has been increasingly challenged by the rising number of applicants per residency spot. This application overburdening across competitive specialties led to several adverse downstream effects, which affected all stakeholders. While several changes and proposals were made to rectify the issue of application overburdening, the 2020-2021 ERAS Match Cycle finally saw several competitive specialties, including otolaryngology and urology, utilize a new system of supplemental residency application based on preference signals/tokens. These tokens permit applicants to electronically signal a select number of programs in a specialty of choice, with the program reviewing the application now cognizant that they have been signaled, i.e., the applicant has chosen to use up a limited set of signals for their program. Initial results from otolaryngology and urology, as described in this article, indicated the value of this new system to both applicants and educators. Given the favorable outcomes and broader uptake of the system among other specialties, the field of neurosurgery adopted the utilization of the ERAS-based program signaling and geographic preference for the first time for the 2022-2023 Residency Application Cycle and later opted to continue them for the 2023-2024 and 2024-2025 cycles. For the 2024-2025 Match Cycle, neurosurgery applicants have 25 signals, i.e., a "high-signal" approach, where non-signaled programs have a low interview conversion rate. This literature review discusses the rationale behind the change, the outcomes of other competitive specialties from prior cycles, the evolving nature of the change, and the potential impact on applicants and programs. As we describe in this review, signaling may potentially represent a surrogate form of an application cap. Other considerations relate to cost savings for both applicants and programs from a high-signal approach in neurosurgery. These modifications represent a foundational attempt to alleviate the application overburdening and non-holistic review in the residency application process, including for neurosurgery. While these changes have been a welcomed addition for all stakeholders in residency match cycles so far, further prospectively directed surveys along with qualitative research studies are warranted to better delineate the downstream impact of these changes and guide further optimization of the application system.

Preference signaling for urologists (including Flush Day)

 It's been very interesting for me to watch the growth of preference signaling to combat congestion in applications and interviews, from its beginnings in the job market for new Ph.D. economists, to its spread through the medical specialty job markets hosted by the National Resident Matching Program (NRMP), and on to medical specialties that run their own matches, such as urology.

So, if you want to be a urologist, take note of the signaling deadline today:

The Society of Academic Urologists Resident Match Program 

Applicant Preference Signaling Deadline is today, September 16, 2024

The SAU has a good deal of supporting material on their site. 

Here's a summary of their interview and offer procedures: Interview Offer Summary (which includes an appropriately named Flush Day for finalizing interviews).

And here's their page on Preference Signaling, linking to this Overview, FAQs, and webinar.

The SAU invites applicants to send up to 30 signals, and encourages them to include programs that know them well on their signaling list.  So (as I've remarked in previous posts) I think this is likely to work as a soft cap on applications.

HT: Mike Rees

Signaling for medical residencies: the first few years

Two papers report on signaling in Otolaryngology and Orthopaedic surgery.

 The Otolaryngology societies have a summary of the current state of affairs that's worth hearing. A number of specialties (including Oto) allow many signals, and these seem to be acting as a soft cap on applications, rather than as a signal of special interest as in specialties that (like Economics) allow only a small number of signals.

Preference Signaling in Otolaryngology—Past, Present, and Future: A Comment From the Society of University Otolaryngologists (SUO), Association of Academic Departments in Otolaryngology (AADO), and the Otolaryngology Program Directors Organization (OPDO)  by Steven D. Pletcher MD, Bradley F. Marple MD, David J. Brown MD, The Laryngoscope Early View,  First published: 04 July 2024  https://doi.org/10.1002/lary.31613

"The year 2020 was a year of change. The residency application process, already suffering from spiraling application numbers,1 now faced the COVID-19 pandemic with a loss of away rotations and apprehension about virtual interviews. In the face of change, the Otolaryngology Program Directors Organization Council (OPDO) approached the leadership of the Association of Academic Departments in Otolaryngology (AADO) and the Society of University Otolaryngologists (SUO) with a recommendation to implement preference signaling. This system, originally described in the economics PhD marketplace,2 allows students a set number of signals (Otolaryngology used 5 in its inaugural year) to send to programs of particular interest. 

...

"Following the lead of Otolaryngology, Urology, General Surgery, Internal Medicine, and Dermatology implemented preference signaling the following year. Since that time, signaling has grown exponentially and is now utilized in the residency application process of nearly every specialty. 

...

"In the 2024-2025 residency application cycle, the evolution of preference signaling continues. Building on Otolaryngology's experience, in the 2023 application cycle Orthopaedic Surgery implemented a high-signal approach, providing applicants with 30 signals. This transition shows promise for reversing the vexing problem of spiraling application numbers—“Big Signaling” has now been adopted by Otolaryngology and four additional specialties the majority of whom have shown a 25%–30% decrease in applications submitted per student saving students a combined $2.5 million in application fees alone. Obstetrics and Gynecology has piloted a tiered signaling system, providing three gold and 15 silver signals to their students. 

...

"Because the number of signals received by programs is not publicized, students are unable to reliably target programs where their signals are less likely to be diluted by competing signals. Specialties should consider providing voluntary “signal cohort” (i.e., my program received between 75 and 100 signals in the 2024 application cycle) data to help applicants make more informed signal decisions and programs with low signal numbers will likely attract additional candidates. 

...

"One of the key statistics to guide applicants in high signal specialties is the interview offer rate for non-signal applications: this helps define the value of applications beyond the set number of signals."

########

Preference Signaling in the Orthopaedic Surgery Match: Applicant and Residency Program Attitudes, Behaviors, and Outcomes, by Guthrie, Stuart Trent MD, FAOA1,a; Dagher, Tanios BSE2; Essey-Stapleton, Jodi MS, MEd3; Balach, Tessa MD, FAOA2,  JBJS Open Access 9(2):e23.00146, April-June 2024. | DOI: 10.2106/JBJS.OA.23.00146

"In the first year of preference signaling, applicants reported applying to 16% fewer programs than if preference signaling had not been available. These results align with AAMC data, which report applications per program dropping 17.4% (from an average of 639.6 to 774.6), and applications per applicant dropping to 76.9 from 86.07,8. Further changes could occur in future cycles as students become more accustomed to the influence of signaling on their application."

The Path to a Match for Interventional Cardiology Fellowships

The Society for Cardiovascular Angiography & Interventions has started a fellowship match, and here's an article describing the familiar marketplace failure that led to that decision, involving unraveling of application, interview and appointment dates, with the resulting congestion and exploding offers, and the process of reaching sufficient consensus to move to a centralized match ( to be run by the NRMP).

The Path to a Match for Interventional Cardiology Fellowship: A Major SCAI Initiative  by Douglas E. Drachman MD, FSCAI (Chair) a, Tayo Addo MD b, Robert J. Applegate MD, MSCAI c, Robert C. Bartel MSc, CAE d, Anna E. Bortnick MD, PhD, MSc, FSCAI e, Francesca M. Dea d, Tarek Helmy MD, MSCAI f, Timothy D. Henry MD, MSCAI g, Adnan Khalif MD, FSCAI h, Ajay J. Kirtane MD, SM, FSCAI i, Michael Levy MD, MPH, FSCAI j, Michael J. Lim MD, MSCAI k, Ehtisham Mahmud MD, MSCAI l, Nino Mihatov MD, FSCAI m, Sahil A. Parikh MD, FSCAI i, Laura Porter CMP d, Abhiram Prasad MD n, Sunil V. Rao MD, FSCAI o, Louai Razzouk MD, MPH, FSCAI o, Samit Shah MD, PhD, FSCAI p, Adhir Shroff MD, MPH, FSCAI q, Jacqueline E. Tamis-Holland MD, FSCAI r, Poonam Velagapudi MD, FSCAI s, Fredrick G. Welt MD, FSCAI t, J. Dawn Abbott MD, FSCAI (Co-Chair), Journal of the Society for Cardiovascular Angiography & Interventions, in press.

"Abstract: The field of interventional cardiology (IC) has evolved dramatically over the past 40 years. Training and certification in IC have kept pace, with the development of accredited IC fellowship training programs, training statements, and subspecialty board certification. The application process, however, remained fragmented with lack of a universal process or time frame. In recent years, growing competition among training programs for the strongest candidates resulted in time-limited offers and high-pressure situations that disadvantaged candidates. A grassroots effort was recently undertaken by a Society for Cardiovascular Angiography & Interventions task force, to create equity in the system by establishing a national Match for IC fellowship. This manuscript explores the rationale, process, and implications of this endeavor."

"over the past several years program directors and candidates found that the process has devolved, with wide variation in application timelines and on-the-spot offers, which disadvantage candidates and programs looking to interview a range of applicants.

"The pressures and unfair features of the existing system were further fueled by the transition to virtual interviews related to the COVID-19 pandemic. With logistics of travel no longer a consideration, programs could commence interviews nearly immediately after the applications became available. This led to more candidates being interviewed in rapid succession, and a system evolved in which programs quickly assessed candidates, offered positions, and applied pressure for candidates to accept offers or be passed over for other candidates.

"In response to the shortcomings of the current system, members of Society for Cardiovascular Angiography & Interventions (SCAI) were inspired to lead a grassroots educational campaign to organize IC program directors and the broader interventional community to commit to a regulated “Match” process under the established National Resident Match Program (NRMP). This manuscript provides an account of how this process unfolded and how a Match for IC fellowship was ultimately created.

...

"From the applicant’s perspective, the lack of a structured timeline for the application process required candidates to make career decisions early in the first year of cardiovascular disease training and to compose their application materials 2 years in advance of starting IC training. With ERAS open to application submission in the fall of the second year for the December release to programs, fellows had limited time on clinical rotations to determine their interest and aptitude for IC. Additionally, letters of recommendation, written at this early stage, risked not being fully reflective of each candidate’s capacity to improve and develop the technical skills and clinical knowledge important for success in the field. There were other disadvantages to candidates in the existing system. Fellows at programs with an IC fellowship had an advantage of securing an internal spot but were often pressured to limit their exploration of the opportunities at other programs, potentially disadvantaging them in the long term.

"Another problem with the existing system was that the pressure to recruit candidates on a tight timeline limited the opportunity to interview applicants from a wide variety and diversity of programs, potentially reducing the ability to recruit underrepresented candidates from varied programs. Despite an overall increase in the diversity of physicians entering the workforce,11 there has been little change in the applicant pool for IC over the years, with fewer than 5% of applicants self-reporting as Black race or Hispanic ethnicity and only 10% identifying as women.12

"Competition among the programs, each vying for the seemingly strongest candidates, degenerated into a system that favored quick decision-making on the part of programs to offer positions as early as possible. The influence of the COVID-19 pandemic in 2020 and 2021 negatively impacted an already high-pressure application process, compounding its many weaknesses.13 Fellowship interviews were hosted virtually rather than in person, which enabled candidates to interview at a greater number of programs without the need to travel. In addition, the virtual format accelerated the tempo of an application process that was already felt to be too fast, resulting in an increase in so-called “exploding offers”—offers that required the accepted candidate to respond within a very short timeframe or risk losing the offer. This practice placed significant pressure on candidates to make quick decisions, often forcing them to determine whether to accept the offer from 1 institution before having the opportunity to participate in interviews with—let alone see and evaluate—other programs or fully understand the ramifications of accepting an offer on their personal lives. At the same time, the accelerated timetable left many programs scrambling to identify applicants, as the number of available candidates diminished rapidly due to applicants accepting time-sensitive, exploding offers.

...

"As with other national efforts of this magnitude, the path to develop consensus in favor of a Match was not without challenges. There were several program directors around the country who strongly opposed the institution of a Match. These were well-regarded academicians and clinician educators who expressed very sincere concerns about the impact on fellows in their programs. The members of the SCAI Match Task Force addressed as many concerns as possible, providing the information necessary for each program director to make the best decision for their institution. A minority of program directors remained opposed to the initiative or did not engage with Task Force members despite multiple attempts to be contacted.

"The Match campaign proved highly effective, and by November 2022, the 75% threshold of programs and positions to implement the Match was met

...

"As the sponsor of the Match, SCAI considered the pros and cons of the “All In Policy,” where registered programs must attempt to fill all ACGME positions at the program through the Match.15,16 SCAI opted out of the “All In Policy” to allow programs to have flexibility for unique situations that require commitment to a candidate outside of the Match. 

...

"As a result of the successful implementation of the Match in IC, the first Match cycle for incoming IC fellows will open in the summer of 2024. Individuals eligible to apply include cardiovascular disease fellows in their third or final year of training and graduates who have completed fellowship and are in clinical practice. This class will start IC training in July 2025"

NRMP Board of Directors (2020-2024), and a 40th anniversary

 This bit of glass marks the end of my term on the National Resident Matching Program (NRMP) Board of Directors.

One of the issues that consumed a lot of attention during my term is discussed in this post:

Friday, April 21, 2023

Transition from medical school to residency: defending the parts that work well (namely the NRMP Resident Match)

And here are all my posts about residents and fellows, going back to the beginning of this blog in 2008. (It's been interesting watching medical specialties begin to develop signaling in ways  reminiscent of signaling in the Economics job market, to deal with congestion of interviews and applications.*)

My first paper dealing explicitly with The Match suddenly seems to have been published 40 years ago:

Roth, A.E. "The Evolution of the Labor Market for Medical Interns and Residents: A Case Study in Game Theory", Journal of Political Economy, Vol. 92, 1984, 991‑1016. http://web.stanford.edu/~alroth/papers/evolut.pdf 

And the main report (with Elliott Peranson) of our redesign of The Match is now a quarter of a century old:

Roth, A.E. and E. Peranson, "The Redesign of the Matching Market for American Physicians: Some Engineering Aspects of Economic Design,” American Economic Review, 89, 4, September, 1999, 748-780. https://www.aeaweb.org/articles?id=10.1257/aer.89.4.748

*See yesterday's post for some discussion of market design interventions in job markets.

Signaling in medical residency applications

 We're starting to see descriptive studies of how signals are being used in the labor market for new doctors.  Each medical specialty has chosen to adapt the kinds of signals used in Economics in its own way, with some specialties using only a handful of signals and others eliciting as many as 30.

Here are two papers from a recent issue of Academic Medicine.

Impact of Applicants’ Characteristics and Geographic Connections to Residency Programs on Preference Signaling Outcomes in the Match, by Benjamin, William J. MPH; Lenze, Nicholas R. MD, MPH; Bohm, Lauren A. MD; Thorne, Marc C. MD, MPH; Abraham, Reeni MD; Sepdham, Dan MD; Mihalic, Angela P. MD; Kupfer, Robbi A. MD,  Academic Medicine 99(4):p 437-444, April 2024. | DOI: 10.1097/ACM.0000000000005551

"Abstract

Purpose : To assess the impact of applicant and residency program characteristics on preference signaling outcomes in the Match during the first 2 years of implementation across 6 specialties.

Method : Data were obtained from the Texas Seeking Transparency in Application to Residency survey for applicants applying into otolaryngology during the 2020–2021 and 2021–2022 application cycles and into dermatology, internal medicine (categorical and preliminary year), general surgery, and urology during the 2021–2022 application cycle. The primary outcome was signal yield, defined as the number of interviews at signaled programs divided by the total number of signals sent. Associations with applicant-reported characteristics and geographic connections to residency programs were assessed using Wilcoxon rank sum testing, Spearman’s rank correlation testing, and ordinary least squares regression.

Results : 1,749 applicants with preference signaling data were included from internal medicine (n = 884), general surgery (n = 291), otolaryngology (n = 217), dermatology (n = 147), urology (n = 124), and internal medicine preliminary year (n = 86). On average 60.9% (standard deviation 32.3%) of signals resulted in an interview (signal yield). There was a stepwise increase in signal yield with the percentage of signals sent to programs with a geographic connection (57.3% for no signals vs. 68.9% for 5 signals, P < .01). Signal yield was positively associated with applicant characteristics, such as United States Medical Licensing Exam Step 1 and 2 scores, honors society membership, and number of publications (P < .01). Applicants reporting a lower class rank quartile were significantly more likely to have a higher percentage of their interviews come from signaled programs (P < .01).

Conclusions: Signal yield is significantly associated with geographic connections to residency programs and applicant competitiveness based on traditional metrics. These findings can inform applicants, programs, and specialties as preference signaling grows."

And here are the introductory paragraphs:

"The rising number of residency applications submitted per applicant has led to concerns that programs will not be able to adequately perform a holistic review of all applications and will instead rely on easily reviewed metrics, such as United States Medical Licensing Exam (USMLE) scores, class rank, and medical school reputation.1,2 In addition, COVID-19–related changes to the residency application process, such as the introduction of virtual interviewing and a cap on the number of away rotations medical students can complete, have limited applicants’ ability to informally express their interest in programs.3 Further, there is evidence that, while applying to the maximum number of programs is advantageous at the individual level, it leads to poorer overall results when all applicants follow this practice.2 To address this issue, new systems have been proposed, including personalized application paragraphs,4 program-specific messages,5 and preference signals.4,6,7

"Otolaryngology implemented a preference signaling system in 2021, which was based on theory developed by the American Economic Association (AEA) in 2006. The AEA used a preference signaling system for job market applicants, whereby applicants were allowed to express special interest in particular employers in their applications.8 Results from the AEA program highlighted that preference signals were beneficial to both candidates and employers in a labor market where employers are unable to provide full attention to every application they receive.9 Building off previous economic work, a computer simulation study run on 2014 otolaryngology Match data found that the number of interview invitations improved when applicants provided preferences on their Electronic Residency Application Service application; this result would have benefitted both programs and applicants.2

"Based on this research, the Otolaryngology Program Directors Association formally implemented a preference signaling system during the 2020–2021 application cycle in which applicants were granted 5 “signals” to send to residency programs prior to interviews indicating their strong interest in that program. Each program then received a list of the applicants who had sent them a signal.6,10,11 Data from the 2021 otolaryngology Match were notable for significantly increased interview rates at signaled programs across all levels of applicant competitiveness.10,12 Furthermore, the majority of program directors and applicants strongly supported the continuation of preference signaling.10,11 During the 2021–2022 application cycle, preference signaling pilot programs were implemented in 5 additional specialties: dermatology, internal medicine (categorical), internal medicine preliminary year, surgery (categorical), and urology, with each specialty using 5 signals per applicant, except dermatology, which used 3 signals"

#########

The Relationship Between Program and Applicant Characteristics With Applicant Program Signals in the 2022 Residency Recruitment Cycle: Findings From 3 Specialties, by LaFemina, Jennifer MD; Rosman, Ilana S. MD; Wallach, Sara L. MD; Wise, Paul E. MD; Smink, Douglas S. MD, MPH; Fletcher, Laura PhD, Academic Medicine 99(4):p 430-436, April 2024. | DOI: 10.1097/ACM.0000000000005586

"Abstract

Purpose: Continuing increases in application volume have driven a national dialogue to reform the residency recruitment process. Program signaling allows applicants to express interest in a program at the preinterview stage with the goal of helping programs identify applicants with more genuine interest in their programs. This study explored the relationship between program signals and program and applicant characteristics.

Method: Participating dermatology, general surgery, and categorical internal medicine (IM) programs and applicants of the 2022 supplemental ERAS application (SuppApp) were included. Data from the SuppApp, the MyERAS Application for Residency Applicants (MyERAS), and the 2020 GME Track Survey were used. Cohen’s h was used to determine effect size, and chi-squared was used to determine statistical significance.

Results:There was an uneven distribution of signals to programs, with 25% of programs receiving about half of the signals across all 3 specialties. Programs with larger numbers of both residents and applicants received greater numbers of program signals relative to their program density, although this effect was small (h < 0.50, P < .001). No meaningful differences were seen across genders for any specialty. Only Hispanic applicants in IM sent a higher proportion of signals to programs with more underrepresented in medicine residents than White only applicants (40% vs 26%, h = 0.30, P < .001). Across all specialties, there was a small-to-moderate effect for international medical graduate (IMG) applicants sending a larger proportion of signals to programs with more IMG residents (h < 0.80, P < .001).

Conclusions: This first-year pilot study (i.e., SuppApp) provided initial evidence that supports the feasibility and fairness of program signals in residency selection. As program signals become more common across specialties, future research should continue to evaluate trends in where applicants send signals, and possible relationships between program and application characteristics."

"IMG applicants were more likely to signal programs with a greater proportion of IMG residents. The effect was small in dermatology and increased to moderate in GS and large in IM. In the NRMP’s 2022 Main Residency Match, 11 IMGs (U.S. and non-U.S.) matched into postgraduate year 2 dermatology, representing 2% of positions. This compares to the 10% and 38% IMG Match rate into GS and IM, respectively.21 While at this time, correlation of signal distribution and the likelihood of successfully matching is not available, these findings suggest that in general, IMG applicants sent more signals to programs they knew to be “IMG friendly” (i.e., more likely to accept IMGs), which they could easily identify with tools such as the Residency Explorer Tool22 and the Residency Programs List.23 However, if IMGs continue to send more signals to programs with already higher proportions of IMG residents, this may maintain the status quo or even further restrict the IMG applicant pool all programs are willing to consider during their resident selection process because programs with fewer IMGs will continue to receive a lower proportion of signals from IMG applicants. This could ultimately negatively affect diversity across programs"

Match Day for new doctors

 The 2024 Match for new American doctors was announced yesterday by the NRMP, the National Resident Matching Program. Congratulations to all!

Here are some links:

Advance Data Tables
Match By the Numbers
Press Release

Kidney exchange (and other bits of market design) in the New York Times

 Peter Coy, the veteran New York Times economics columnist, writes about kidney exchange, after an interview/conversation sparked by a recent working paper of mine, Market Design and Maintenance. (He's a rare economic journalist who reads economists' papers.)

Here's his column, published yesterday afternoon:

The Economist Who Helped Patients Get New Kidneys, Feb. 5, 2024, 3:00 p.m. ET, By Peter Coy

He's also a rare interviewer: his column includes the names of more of my coauthors than I can recall in any other interview. In order of appearance: Tayfun Sonmez and Utku Unver, Frank Delmonico, Susan Saidman, Mike Rees (implicitly) when he names Mike's nonprofit Alliance for Paired Kidney Donation, and Elliott Peranson.  Market design is, after all, a team sport.

Here's his concluding paragraph:

"What is it like to straddle the worlds of academia and practice? I asked. “It takes a lot of patience,” he said. “Market design is outward-facing. I learn from trying to persuade people who aren’t economists. It’s a lot of fun also. Sometimes you have to go beyond your completely reliable scientific knowledge.”

########

Earlier post:

Monday, December 11, 2023

Market Design and Maintenance (new NBER working paper, from a conference)

Signaling for residency programs in dermatology, general surgery, and internal medicine

We're starting to see some data from signaling for residency applications.  This paper observes that programs are more likely to interview candidates who send them a signal. (Economists will worry that this reflects which programs are signaled and not just the effect of a signal...)  These three specialties have relatively few signals, more like economics than like Orthopedic Surgery (which has 30 signals).  And the table indicates that more interviews are offered than signals received, so that's another difference from Ortho...)

Rosenblatt, Adena E., Jennifer LaFemina, Lonika Sood, Jennifer Choi, Jennifer Serfin, Bobby Naemi, and Dana Dunleavy. "Impact of Preference Signals on Interview Selection Across Multiple Residency Specialties and Programs." Journal of Graduate Medical Education 15, no. 6 (2023): 702.

"Abstract

"Background Program signaling is an innovation that allows applicants to express interest in specific programs while providing programs the opportunity to review genuinely interested applicants during the interview selection process.

"Objective To examine the influence of program signaling on “selected to interview” status across specialties in the 2022 Electronic Residency Application Service (ERAS) application cycle.

"Methods Dermatology, general surgery-categorical (GS), and internal medicine-categorical (IM-C) programs that participated in the signaling section of the 2022 supplemental ERAS application (SuppApp) were included. Applicant signal data was collected from SuppApp, applicant self-reported characteristics collected from the MyERAS Application for Residency Applicants, and 2020 program characteristics collected from the 2020 GME Track Survey. Applicant probability of being selected for interview was analyzed using logistic regression, determined by the selected to interview status in the ERAS Program Director’s WorkStation.

"Results Dermatology had a 62% participation rate (73 of 117 programs), GS a 75% participation rate (174 of 232 programs), and IM-C an 86% participation rate (309 of 361 programs). In all 3 specialties examined, on average, signaling increased the likelihood of being selected to interview compared to applicants who did not signal. This finding held across gender and underrepresented in medicine (UIM) groups in all 3 specialties, across applicant types (MDs, DOs, international medical graduates) for GS and IM-C, and after controlling for United States Medical Licensing Examination Step 1 scores.

"Conclusions Although there was variability by program, signaling increased likelihood of being selected for interview without negatively affecting any specific gender or UIM group."

Data from future years will be needed to determine how signaling is influencing the distribution of residents to programs.

Applying for medical residencies: a consensus statement from Internal Medicine

 The Alliance for Academic Internal Medicine has released a "consensus statement" with many proposals about application and interview caps, and signaling.

Catalanotti, Jillian S., Reeni Abraham, John H. Choe, Kelli A. Corning, Laurel Fick, Kathleen M. Finn, Stacy Higgins et al. "Rethinking the Internal Medicine Residency Application Process to Prioritize the Public Good: A Consensus Statement of the Alliance for Academic Internal Medicine." The American Journal of Medicine (2023).

It also includes a call for data and analysis:

"AAIM proposes increasing internal medicine program preference signals to 15, using tiered signaling with three “gold” and 12 “silver” signals, and setting an interview cap of 15 in the 2024-2025 recruitment season, with participation by all internal medicine programs. The Alliance recommends that all internal medicine programs participate in ACI. AAIM recommends that programs transparently share information about their use of preference signals and other application screening methods and calls for real-time data analysis to explore impact, inform future iterations and identify potential harms.

"The Alliance calls upon ERAS and NRMP as well as Thalamus® and other interview scheduling platforms to transparently share data, to embrace change, and to perform analyses needed to inform this process. For example, recent modeling with eight years of retrospective NRMP data in OBGYN demonstrated that an early match round may increase the number of “mutually dissatisfied applicant-program pairs” and that a multiple-round match process could introduce potential rewards for gamesmanship, a prime factor addressed by the current process.35 AAIM applauds this analysis and hopes that the new collaboration between ERAS and Thalamus® may provide useful interview data to inform this proposal and further interventions."

And here is reference 35 in that last paragraph, about which I've blogged before.

I Ashlagi, E Love, JI Reminick, AE. Roth

Early vs Single Match in the Transition to Residency: Analysis Using NRMP Data From 2014 to 2021

J Grad Med Educ, 15 (2) (Apr 2023), pp. 219-227, 10.4300/JGME-D-22-00177.1

View in ScopusGoogle Scholar

Transition to residency conference: Oct 5-7

I'll be a panelist at the  the NRMP conference Transition to Residency,  in Boston, Oct 5-7

"The National Resident Matching Program® (NRMP®) will convene its stakeholder conference in Boston this year. The meeting is intended to provide a forum for robust conversation among members of the undergraduate and graduate medical education communities about issues relevant to the transition to residency."

Here's the list of plenary speakers.

Friday, Oct 6, 8:30 – 9:45 AM Plenary I

The Future of the Transition to Residency: Assessing the Impact of Proposed Change

Panelists:

John Combes, MD

Alvin Roth, PhD

Charles (Tom) Thomas, MA, MPhil

Mathematics and Computer Science of Market and Mechanism Design: SLMath introductory workshop (videos)

Last week I gave the opening talk of the week long  Introductory Workshop at SLMath, on Mathematics and Computer Science of Market and Mechanism Design.  Some of the video lectures are now online here (consisting mostly of slides and voice).

My talk introduces the general themes of market design by recounting the history and challenges facing the market for new doctors from 1900 through this year.

Berkeley's Simons Laufer Mathematical Sciences Institute (SLMath), formerly known as the Mathematical Sciences Research Institute (MSRI) has a commanding view of the SF Bay.

Signaling for Orthopaedic surgery residencies

 The Journal of Bone and Joint Surgery reports a small survey about signaling behavior among applicants for orthopaedic surgery residencies.

Deckey, David G., Eugenia Lin, Coltin RB Gerhart, Joseph C. Brinkman, Karan A. Patel, and Joshua S. Bingham. "Decoding the Signals: An Analysis of Preference Signaling in the 2023 Orthopaedic Surgery Residency Match." JBJS Open Access 8, no. 3 (2023).

"While previously used in other specialties, the preference signaling program (PSP) was implemented in the 2022 to 2023 orthopaedic surgery residency application process for the first time. The PSP allowed for 30 signaling tokens to be sent by applicants to programs of their choice to indicate particular interest in a program.

...

"An anonymous electronic survey was emailed to all orthopaedic surgery residency applicants who applied to the authors' institution during the 2022 to 2023 application cycle. The survey was sent after match lists were submitted and closed before the release of match results. 

...

"The survey was completed by 101 applicants. Applicants applied to a mean of 90 programs (range: 10-197) and received an average of 12 interview invitations (range: 0-39). Applicants almost uniformly used all 30 signals, with nearly two-thirds signaling their home programs (65%, 49/76), and nearly all applicants sending signals to programs at which they performed away rotations (95.7%, 88/92). Applicants received a mean of 9 invitations from programs they signaled, compared with 2 invitations from programs they did not signal."

#######

Applicants report sending signals to all the programs that would have been expected to automatically give them interviews even in the absence of a signaling mechanism--namely their home programs and those which they have spent time visiting in 'away rotations.'

In the Economics job market, which may have been the first to introduce signaling, we limited applicants to 2 signals, and advised them not to signal jobs in which they already had well established mutual interests, i.e. not to signal jobs which they felt would interview them without signals. One of the ideas behind the Econ signaling mechanism is that there are many ways applicants can send signals of interest within a network to which they are well connected. Since some applicants are better connected than others, we were offering some signals that could be sent out of network.

It will be interesting to understand if signals to Ortho residencies are helping promote out of network interviews, or are largely strengthening the network connections already established by medical schools (home programs) and away rotations.

OB-GYN doctors will use a new application system to apply to residency programs (but will continue to go through the NRMP resident match)

 Before new doctors can participate in the resident match (by engaging with the NRMP), they first have to apply to residency programs, and arrange interviews.  This process has been experiencing congestion, and the specialty of Obstetrics and Gynecology has now decided to switch application services. 

However, participation in the NRMP will not change: the Association of Professors of Gynecology and Obstetrics (APGO) FAQ states "Obstetrics and gynecology applicants will use the National Residency Match Program (NRMP) for the Match. This new application does not change how the applicant or programs interact with the NRMP Match system."

Medpage Today has the story:

Ob/Gyn Switching to Independent System for Residency Applications— This is the last year ob/gyn will use ERAS, despite helping to pilot the program  by Rachael Robertson, Enterprise & Investigative Writer, MedPage Today August 25, 2023

"Beginning next year, ob/gyn programs will start using an independent system for processing residency applications, rather than the Electronic Residency Application Service (ERAS).

"The joint decision to switch to the new system was made by the American College of Obstetricians and Gynecologists (ACOG), the Association of Professors of Gynecology and Obstetrics (APGO), and the Council on Resident Education in Obstetrics and Gynecology (CREOG). The new system will be managed by Liaison International, which uses "Centralized Application Service (CAS) technology," according to the company's website.

...

"A joint statement on the APGO website opens in a new tab or window

said that the new system "will be user friendly and efficient, less expensive for applicants, and will directly decrease the burdens faced by program directors, program managers, and applicants alike," and "will incorporate the entirety of interview season functions, from application submission, review, interview offers and interviews, to rank list submission."

ACOG explained that the decision to pull the ERAS stemmed from the Right Resident, Right Program, Ready Day One initiativeopens in a new tab or window, noting that the new system is mobile-friendly and "will include immediate fee reduction," as detailed on their FAQ pageopens in a new tab or window.

In response, the Association of American Medical Colleges (AAMC), which runs ERAS, issued a statementopens in a new tab or window attributed to President and CEO David J. Skorton, MD, and Alison J. Whelan, MD, the chief academic officer, saying they were "surprised and dismayed" by the decision. 

...

"Bryan Carmody, MD, of Eastern Virginia Medical School in Norfolk, shared information  opens in a new tab or window

about the change on social media, writing on his blogopens in a new tab or window that ob/gyn program directors helped to pilot ERAS when it was first rolled out in the mid-90s.

Carmody told MedPage Today that he anticipates the biggest downsides will fall on applicants, such as those who want to apply to another specialty in addition to ob/gyn.

"Those applicants will have to use one system to apply to ob/gyn and another to their other specialty," he explained. "The same thing applies to applicants who fail to match. They'll have to use ERAS to apply to another specialty during SOAP [Supplemental Offer and Acceptance Program] since few, if any, ob/gyn positions are typically available."

*************

OB-GYN will continue to employ signaling  (very loosely modeled on the signaling used in the Econ PhD job market, but asking applicants to submit 3 "gold" signals and 15 "silver" signals): 

Program Signaling for OBGYN Residency Application Background and FAQs

**************

Earlier:

Friday, April 21, 2023

Transition from medical school to residency: defending the parts that work well (namely the NRMP Resident Match)

Resident match video from the NRMP

 Different labor markets are organized differently. One difference between the market for new doctors and the markets for new Ph.D.s is that doctors use a centralized clearinghouse, so a lot of things happen at the same time.  Here's a video from the National Resident Matching Program that gives some idea about that.

Interesting development in the transition from medical school to residency: connecting applications and interviews

 The market for new doctors has been suffering from congestion in applications and interviews, in the runup to the resident Match (see recent post with a diagram). The American Association of Medical Colleges runs the main application server, ERAS. A private company called Thalamus runs a growing interview scheduling service. Now they are looking to collaborate.

 Here's  yesterday's press release from Thalamus:

AAMC, Thalamus Announce New Collaboration to Improve Transition to Residency

April 27, 2023 by Team Thalamus

Collaboration will increase transparency and make the residency process easier for applicants and programs  

Washington, D.C., April 27, 2023—Today the AAMC (Association of American Medical Colleges) and Thalamus announced a strategic collaboration to accelerate innovation and ease the transition to residency for medical students, medical schools, and residency programs. The collaboration will combine the AAMC’s long-established leadership in innovation along the continuum from medical school to residency training and continuing medical education with Thalamus’ market-leading product and software development expertise. 

“The transition from undergraduate medical education to graduate medical education is a critical period in any learner’s journey to becoming a physician,” said David J. Skorton, MD, AAMC president and CEO. “We know the community is seeking enhanced tools and integrated services that better support application and recruitment processes. We listened, we have made improvements, and, with Thalamus, we are excited to make this vision a reality.” 

The organizations will collaborate to leverage their data, technology, and expertise to transform the medical residency and fellowship recruitment processes for applicants and programs. Their efforts will focus on increasing transparency, supporting equity through holistic review, and improving the learner experience by consolidating the fragmented interview management process. 

“We are thrilled to be collaborating with the AAMC to provide a comprehensive solution that will streamline graduate medical education recruitment processes,” said Jason Reminick, MD, MBA, MS, CEO and founder of Thalamus. “But even more, we are looking forward to building new and innovative tools that improve the experience, are cost-effective, and leverage data for the benefit of the medical education community and the advancement of our collective missions.” Dr. Reminick applied to residency in 2012 during an eventful recruitment season disrupted by Hurricane Sandy. “I’m particularly excited to provide applicants with a comprehensive platform to manage their interview season.” 

The collaboration between the AAMC and Thalamus will enable data-sharing and innovative research that will benefit the undergraduate to graduate medical education community and advance both organizations’ missions. The initiative also demonstrates the commitment of both organizations to addressing the concepts and themes outlined in the 2021 report from the Coalition for Physician Accountability’s Undergraduate Medical Education-Graduate Medical Education Review Committee.

In recent years, the AAMC has completed significant in-depth research and upgraded technology to enhance the Electronic Residency Application Service® (ERAS®) suite of application and selection tools, such as updating the MyERAS® application content, building analytics tools for institutions, and partnering on collaborative research initiatives. Thalamus has completed unique research related to the physician workforce, including how geography influences The Match® and specialty-specific interview practices. The Thalamus technology will continue the upgrade of the ERAS suite of application and selection tools. The AAMC and Thalamus remain committed to future innovations that will enable the ERAS program to continue to evolve faster and better. 

Beginning in June 2023, all ERAS residency and fellowship programs will receive complimentary access to Thalamus’ leading interview management platform, Thalamus Core and Itinerary Wizard, as well as Cerebellum, a novel data and analytics dashboard to assess recruitment outcomes, specifically from a diversity, equity, inclusion, and geographic perspective. Programs may also elect to purchase Thalamus’ video interview platform and Cortex, its technology-assisted holistic application review and screening platform. 

According to AAMC data, the U.S. is expected to experience a shortage of up to 124,000 physicians by 2034. Given the burnout and other challenges to the health care system caused by the COVID-19 pandemic, the AAMC and Thalamus look to use their collective expertise to promote a diverse and representative workforce that will enhance health care and patient outcomes. 

The data and research the AAMC and Thalamus have amassed to identify resident, fellow, and physician recruitment trends can potentially have a major impact on diversity in medicine and begin to address several well-established and longstanding systemic challenges. These efforts will support not only the application and selection processes in graduate medical education but also aim to improve the experiences of the U.S. physician workforce over the long term. 

Related Resources 

• Frequently Asked Questions 
• AAMC ERAS Collaboration Page
• Thalamus Collaboration Page

###

Note for editors: Leaders from the AAMC and Thalamus are available to speak with media about this new collaboration and what it means for residency programs and applicants. 

The AAMC (Association of American Medical Colleges) is a nonprofit association dedicated to improving the health of people everywhere through medical education, health care, medical research, and community collaborations. Its members are all 157 U.S. medical schools accredited by the Liaison Committee on Medical Education; 13 accredited Canadian medical schools; approximately 400 teaching hospitals and health systems, including Department of Veterans Affairs medical centers; and more than 70 academic societies. Through these institutions and organizations, the AAMC leads and serves America’s medical schools and teaching hospitals and the millions of individuals across academic medicine, including more than 193,000 full-time faculty members, 96,000 medical students, 153,000 resident physicians, and 60,000 graduate students and postdoctoral researchers in the biomedical sciences. Following a 2022 merger, the Alliance of Academic Health Centers and the Alliance of Academic Health Centers International broadened the AAMC’s U.S. membership and expanded its reach to international academic health centers. Learn more at aamc.org. 

Thalamus is the premier, cloud-based interview management platform designed specifically for application to Graduate Medical Education (GME) training programs. The software streamlines communication by eliminating unnecessary phone calls/emails allowing applicants to book interviews in real-time, while acting as a comprehensive applicant tracking system for residency and fellowship programs. Thalamus provides comprehensive online interview scheduling and travel coordination via a real-time scheduling system, video interview platform, AI application screening/review tool (Cortex) providing technology-assisted holistic review, and first-in-class DEI-focused analytics dashboard (Cerebellum). Featured nationally at over 300+ institutions and used by >90% of applicants, Thalamus is the most comprehensive solution in GME interview management. For more information on Thalamus, please visit https://thalamusgme.com or connect with us on LinkedIn, Facebook, Instagram, Twitter, or YouTube. 

Transition from medical school to residency: defending the parts that work well (namely the NRMP Resident Match)

This post is about a recently published paper concerning the design of the market for new doctors in the U.S.  But it will require some background for most readers of this blog.   The short summary is that the market is experiencing problems related to congestion, and one of the proposals to address these problems was deeply flawed, and would have reduced market thickness and caused substantial direct harm to participants if implemented, and created instabilities that would likely have caused indirect harms to the match process in subsequent years. But this needed to be explained in the medical community, since that proposal was being  very actively advocated.

For those of you already steeped in the background, you can go straight to the paper, here.

Itai Ashlagi, Ephy Love, Jason I. Reminick, Alvin E. Roth; Early vs Single Match in the Transition to Residency: Analysis Using NRMP Data From 2014 to 2021. J Grad Med Educ 1 April 2023; 15 (2): 219–227. doi: https://doi.org/10.4300/JGME-D-22-00177.1

If the title doesn't remind you of the vigorous advocacy for an early match for select positions, here is some of the relevant back story.

The market for new doctors--i.e. the transition from medical school to residency--is experiencing growing pains as the number of applications and interviews has grown, which imposes costs on both applicants and residency programs.  

Below is a schematic of that process, which begins with applicants submitting applications electronically, which makes it easy to submit many.  This is followed by residency programs inviting some of their applicants to interview. The movement to Zoom interviews has made it easier to have many interviews also (although interviews were multiplying even before they moved to Zoom).  

After interviews, programs and applicants participate in the famous centralized clearinghouse called The Match, run by the NRMP. Programs and applicants each submit rank order lists (ROLs) ranking those with whom they interviewed, and a deferred acceptance algorithm (the Roth-Peranson algorithm) produces a stable matching, which is publicly announced on Match Day. (Unmatched people and positions are invited into a now computer-mediated scramble, called SOAP, and these matches too are announced on Match  Day.)

The Match had its origins as a way to control the "unraveling" of the market into inefficient bilateral contracts, in which employment contracts were made long before employment would commence, via exploding offers that left most applicants with very little ability to compare options.  This kind of market failure afflicted not only the market for new physicians (residents), but also the market for later specialization (as fellows). Consequently, over the years, many specialties have turned to matching for their fellowship positions as well.

  The boxes in brown in the schematic are those that constitute "The Match:" the formulation and submission of the ROLs, and the processing of these into a stable matching of programs to residents.  Congestion is bedeviling the parts in blue.

The boxes colored brown are 'The Match' in which participants formulate and submit rank order lists (ROLs), after which a deferred acceptance algorithm produces a stable matching of applicants to programs, which is accepted by programs and applicants on Match Day. The boxes in blue, the applications and interviews that precede the Match, are presently suffering from some congestion.  Some specialties have been experimenting with signals (loosely modeled on those in the market for new Economics PhDs, but implemented differently by different medical specialties).

The proposal in question was to divide the match into two matches, run sequentially, with the first match only allowing half of the available positions to be filled.  The particular proposal was to do this first for the OB-GYN specialty, thus separating that from the other specialties in an early match, with only half of the OB-GYN positions available early.

This proposal came out of a study funded by the American Medical Association, and it was claimed, without any evidence being offered, that it would solve the current problems facing the transition to residency.  Our paper was written to provide some evidence of the likely effects, by simulating the proposed process using the preferences (ROLs) submitted in previous years.  

The results show that the proposal would largely harm OB-GYN applicants by giving them less preferred positions than they could get in a traditional single match, and that it would create instabilities that would encourage strategic behavior that would likely undermine the successful operation of the match in subsequent years.

Itai Ashlagi, Ephy Love, Jason I. Reminick, Alvin E. Roth; Early vs Single Match in the Transition to Residency: Analysis Using NRMP Data From 2014 to 2021. J Grad Med Educ 1 April 2023; 15 (2): 219–227. doi: https://doi.org/10.4300/JGME-D-22-00177.1

Abstract:

"Background--An Early Result Acceptance Program (ERAP) has been proposed for obstetrics and gynecology (OB/GYN) to address challenges in the transition to residency. However, there are no available data-driven analyses on the effects of ERAP on the residency transition.

"Objective--We used National Resident Matching Program (NRMP) data to simulate the outcomes of ERAP and compare those to what occurred in the Match historically.

"Methods--We simulated ERAP outcomes in OB/GYN, using the de-identified applicant and program rank order lists from 2014 to 2021, and compared them to the actual NRMP Match outcomes. We report outcomes and sensitivity analyses and consider likely behavioral adaptations.

"Results--Fourteen percent of applicants receive a less preferred match under ERAP, while only 8% of applicants receive a more preferred match. Less preferred matches disproportionately affect DOs and international medical graduates (IMGs) compared to US MD seniors. Forty-one percent of programs fill with more preferred sets of applicants, while 24% fill with less preferred sets of applicants. Twelve percent of applicants and 52% of programs are in mutually dissatisfied applicant-program pairs (a pair in which both prefer each other to the match each received). Seventy percent of applicants who receive less preferred matches are part of a mutually dissatisfied pair. In 75% of programs with more preferred outcomes, at least one assigned applicant is part of a mutually dissatisfied pair.

"Conclusions--In this simulation, ERAP fills most OB/GYN positions, but many applicants and programs receive less preferred matches, and disparities increase for DOs and IMGs. ERAP creates mutually dissatisfied applicant-program pairs and problems for mixed-specialty couples, which provides incentives for gamesmanship."

************

I'm hopeful this paper will effectively contribute to the ongoing discussion of how, and how not, to modify the design of the whole process of transition to residency with an aim to fixing the parts that need fixing, without damaging the parts that work well, i.e. while doing no harm. 

(Signaling will likely continue to play a role in this.)