Standardized testing returns to college admissions

 Standardized admissions exams are making a comeback:

Elite French university revives entrance exam to combat AI fears. Sciences Po reverses changes intended to make admissions more egalitarian, saying ChatGPT and private tutoring mean essays are no longer ‘objective’  by Maurício Alencar, Wednesday October 23 2024, 12.00pm BST, The Times 

"Sciences Po’s new director, Luis Vassy, said standardised testing would make admissions more “objective”, despite one of his predecessors, Frédéric Mion, dropping the exam in an attempt to make the process more accessible to poorer students who had to pay for travel into Paris to sit the test."

%%%%%%%%%%%

Here's the story from the Harvard Gazzette (in April):

Harvard announces return to required testing. Leading researchers cite strong evidence that testing expands opportunity

"Students applying to Harvard College for fall 2025 admission will be required to submit standardized test scores, the Faculty of Arts and Sciences announced ... This new policy will be applied to the Class of 2029 admissions cycle and will be formally assessed at regular intervals. 

...

"“Standardized tests are a means for all students, regardless of their background and life experience, to provide information that is predictive of success in college and beyond,” she said. “Indeed, when students have the option of not submitting their test scores, they may choose to withhold information that, when interpreted by the admissions committee in the context of the local norms of their school, could have potentially helped their application. In short, more information, especially such strongly predictive information, is valuable for identifying talent from across the socioeconomic range.”

 

"In research published last year, Harvard Professors Raj Chetty and David J. Deming and co-author John N. Friedman used data from more than 400 institutions and about 3.5 million undergrads per year to better understand socioeconomic diversity and admissions. Standardized tests emerged as an important tool to identify promising students at less-well-resourced high schools, particularly when paired with other academic credentials.

 

“Critics correctly note that standardized tests are not an unbiased measure of students’ qualifications, as students from higher-income families often have greater access to test prep and other resources,” said Chetty, the William A. Ackman Professor of Public Economics and director of Opportunity Insights. “But the data reveal that other measures — recommendation letters, extracurriculars, essays — are even more prone to such biases. Considering standardized test scores is likely to make the admissions process at Harvard more meritocratic while increasing socioeconomic diversity."

Many preference signals as a soft cap on number of applications in medical residency matching

 Here's a review article on matching for medical residents,  with particular attention to neurosurgery, in the Cureus Journal of Medical Science.  In specialties that (like neurosurgery) allow applicants to send many signals, many applicants signal to and match with programs with which they have some prior connection.

Ozair, Ahmad, Jacob T. Hanson, Donald K. Detchou, Matthew P. Blackwell, Abigail Jenkins, Marianne I. Tissot, Umaru Barrie et al. "Program Signaling and Geographic Preferences in the United States Residency Match for Neurosurgery." Cureus 16, no. 9 (2024).

Abstract: Postgraduate residency training has long been the cornerstone of academic medicine in the United States. The Electronic Residency Application Service (ERAS), managed by the Association of American Medical Colleges (AAMC), is the central residency application platform in the United States for most clinical specialties, with the National Residency Matching Program (NRMP) being the algorithm for matching residency programs with applicants. However, the determination of the best fit between ERAS applicants and programs has been increasingly challenged by the rising number of applicants per residency spot. This application overburdening across competitive specialties led to several adverse downstream effects, which affected all stakeholders. While several changes and proposals were made to rectify the issue of application overburdening, the 2020-2021 ERAS Match Cycle finally saw several competitive specialties, including otolaryngology and urology, utilize a new system of supplemental residency application based on preference signals/tokens. These tokens permit applicants to electronically signal a select number of programs in a specialty of choice, with the program reviewing the application now cognizant that they have been signaled, i.e., the applicant has chosen to use up a limited set of signals for their program. Initial results from otolaryngology and urology, as described in this article, indicated the value of this new system to both applicants and educators. Given the favorable outcomes and broader uptake of the system among other specialties, the field of neurosurgery adopted the utilization of the ERAS-based program signaling and geographic preference for the first time for the 2022-2023 Residency Application Cycle and later opted to continue them for the 2023-2024 and 2024-2025 cycles. For the 2024-2025 Match Cycle, neurosurgery applicants have 25 signals, i.e., a "high-signal" approach, where non-signaled programs have a low interview conversion rate. This literature review discusses the rationale behind the change, the outcomes of other competitive specialties from prior cycles, the evolving nature of the change, and the potential impact on applicants and programs. As we describe in this review, signaling may potentially represent a surrogate form of an application cap. Other considerations relate to cost savings for both applicants and programs from a high-signal approach in neurosurgery. These modifications represent a foundational attempt to alleviate the application overburdening and non-holistic review in the residency application process, including for neurosurgery. While these changes have been a welcomed addition for all stakeholders in residency match cycles so far, further prospectively directed surveys along with qualitative research studies are warranted to better delineate the downstream impact of these changes and guide further optimization of the application system.

Preference signaling for urologists (including Flush Day)

 It's been very interesting for me to watch the growth of preference signaling to combat congestion in applications and interviews, from its beginnings in the job market for new Ph.D. economists, to its spread through the medical specialty job markets hosted by the National Resident Matching Program (NRMP), and on to medical specialties that run their own matches, such as urology.

So, if you want to be a urologist, take note of the signaling deadline today:

The Society of Academic Urologists Resident Match Program 

Applicant Preference Signaling Deadline is today, September 16, 2024

The SAU has a good deal of supporting material on their site. 

Here's a summary of their interview and offer procedures: Interview Offer Summary (which includes an appropriately named Flush Day for finalizing interviews).

And here's their page on Preference Signaling, linking to this Overview, FAQs, and webinar.

The SAU invites applicants to send up to 30 signals, and encourages them to include programs that know them well on their signaling list.  So (as I've remarked in previous posts) I think this is likely to work as a soft cap on applications.

HT: Mike Rees

Tattoos

 Tattoos used to signal something different than they do today.

The NYT has the story.

The Price of Getting Inked. Whether it’s the expense of getting tattooed or the cost to have one removed, Americans are paying for their ink. By Julia Rothman and Shaina Feinberg

"Once considered countercultural — something for sailors and misfits — tattoos are now culturally ubiquitous: Nearly one-third of American adults have at least one, according to a survey by Pew Research.

...

"The global tattoo market, which currently brings in about $2.2 billion, is expected to grow to more than $4 billion by 2032, according to Fortune Business Insights, a market research firm. There are over 20,000 tattoo parlors in the United States."

Preference signaling in the Political Science job market

 There is now signaling in the Political Science job market:  They allow job candidates to send up to three signals: here's the APSA's page with frequently asked questions about signaling.

Aside from the procedures APSA members should use, their site offers some general comments about signaling. Some excerpts:

What is the history of signaling as applied to specialized labor markets like that for political science PhDs?

 The American Economics Association (AEA) Ad Hoc Committee on the Job Market, including Roth, applied his research to the economics PhDs job market over 15 years ago. Since the 1970s, the job market for economists has been organized around the AEA and related associations’ annual meeting (called the Allied Social Sciences Associations, or the “ASSA”) in January and since 2006, economists have utilized signaling for job interviews that take place at the ASSA. For those who don’t secure a position there, a job “scramble” that is more public takes place later in the spring. These events act as clearinghouses to clear congestion resulting from candidates applying for almost all positions due to their need to secure a position, and also prevents “unraveling,” where employers make offers earlier and earlier each year to get the best possible candidate they can.[1] The AEA Ad Hoc Committee on the Job Market offers detailed advice on signaling for (economics) PhDs going on the job market here.

Why would the political science job market benefit from signaling?

The number of political science job postings per PhD on the job market has decreased considerably since 2011, when there were two job openings for each candidate as it recovered from the Great Recession.[1] After the COVID pandemic, the higher education job market seems to be showing signs that it is recovering more slowly, and political science is no exception to this trend. The behavior of the higher education job markets runs counter to the general labor market trend in the US where jobs are currently outnumbering jobseekers.[2]  Currently there are almost two candidates per job opening on APSA eJobs.[3] The political science job market is also congested in 2023. In the last few years, candidates have reported applying for upwards of 25-50, more than 51, or more than 100 positions in a job market cycle, making the possibilities for matches endless.[4]

After the job market at APSA’s Annual Meeting in September, the job market for political science PhDs is decentralized and thin, lacking a centralizing timeframe, location, or deadline, with the only way for candidates to let employers know they are interested is to contact the hiring committee or send in an application for the position. Jobs are being posted earlier each year to secure the best candidates by utilizing the APSA Annual Meeting as a job market clearinghouse. For the remainder of the academic year, those participating in the job market lack any kind of intentionally designed mechanism to structure their path to a job placement before the next academic year..[5] For all these reasons: a surplus of jobseekers, market congestion (applying to all jobs), market unraveling (earlier and earlier offers), the political science job market would benefit from being approached with the market design tools that have been shown to alleviate these inefficiencies.

Citations

How will signaling benefit candidates and employers at APSA Annual Meeting/ Interview Services?

 Due to job market congestion, the job market centralized around the APSA Annual Meeting/ Interview Services would benefit from a mechanism for candidates and employers to be able to decipher meaningful interest more easily from candidates for positions, and to allow institutions to view candidates more equally across characteristics outside of institutional prestige, rank, or publication records. Due to the large number of candidates relative to jobs, we believe signaling can increase the number of matches facilitated, decrease the number of applications both written and submitted by candidates and reviewed by hiring committees, and decrease the stress of all participants in the market. We have created a pilot program to beta test how signaling will work for employers and candidates utilizing Interview Services at the Annual Meeting.

Interviews, flyouts and offers in the job market for new PhD economists, 2024-25

 We may still be in a period of disequilibrium in the conduct of the market for new economists, as Zoom interviews have replaced in-person interviews at the annual January ASSA meetings.  In an attempt at maintaining market thickness, the AEA has reaffirmed its guidelines, as discussed in the email below that went out to AEA members yesterday

AEA Guidance on the 2024-25 Economics Job Market Cycle

Date: August 5, 2024 To: Members of the American Economic Association From: Peter L. Rousseau, Secretary-Treasurer The AEA Executive Committee, in conjunction with its Committee on the Job Market, recognizes that it is to the benefit of the profession if the job market for economists is thick, with many employers and job candidates participating in the same stages at the same time. Moreover, the AEA's goals of diversity, equity, and inclusion are fostered by having a timeline that remains widely known and accepted, ensuring that candidates can correctly anticipate when each stage will occur. The AEA has a role to establish professional norms, which includes ensuring fair treatment of job candidates, including that they have enough time to consider job offers. With these goals in mind, and in light of inquiries from both job candidates and employers about how to proceed, the AEA asks that departments and other employers consider the following timeline for initial interviews and “flyouts” in the upcoming job cycle (2024-25). Timing of interview invitations The AEA suggests that employers wait to extend interview invitations until the day after job market signals are transmitted to employers (planned for December 4). Rationale: the AEA created the signaling mechanism to reduce the problem of asymmetric information and allow job candidates to credibly signal their interest to two employers. The AEA asks that employers wait to extend interview invitations until those signals have been transmitted, and to use that information to finalize their set of candidates to interview. This helps the job market in several ways: it reduces the problem of imperfect information, it helps ensure a thick market at each stage, and it promotes the AEA’s goals of diversity, equity, and inclusion. Job candidates from historically under-represented groups may lack informal professional networks and thus may especially rely on the signals to convey their interest. Waiting to review the signals before issuing invitations promotes a fairer, more equitable process. We also ask that all employers indicate on EconTrack when they have extended interview invitations. This allows candidates to learn about the status of searches without visiting websites posting crowd-sourced information and potentially inappropriate other content. Timing of interviews Initial interviews may take place any time after the AEA signals are sent (planned for December 4). The AEA recommends that all initial interviews take place virtually (e.g., by Zoom). We suggest that interviews not take place during the ASSA meeting itself (January 3-5, 2025). Rationale: In the past, interviews were conducted in-person at the ASSA meetings. This promoted thickness of the market, because most candidates and employers were present at the meetings but had the disadvantage of precluding both job candidates and interviewers from fully participating in sessions. Initial job interviews went online during COVID, and feedback indicated that the benefits of virtual first-round interviews (e.g., low monetary cost, zero cost in travel time, scheduling flexibility, convenience) outweighed the limitations (e.g., less rich interaction). We ask that interviews NOT take place during the ASSA meetings (January 3-5, 2025) in order to allow job candidates and interviewers to participate in the conference. Timing of flyouts Flyouts have historically happened at times appropriate for the employer, and the AEA sees no reason to suggest otherwise. We ask that all employers indicate on EconTrack when they have extended flyout invitations. Unlike with interviews, the AEA does not take a position on whether flyouts should be virtual or in-person. Timing of offers and “exploding” offers In order to ensure that the job market remains sufficiently synchronized and thick, and that candidates have a chance to compare offers, the AEA recommends that employers leave job offers open (i.e. do not require candidates to accept or decline) until at least January 31. The AEA also strongly recommends that employers give candidates at least two weeks to consider their job offer. We recognize that offers made late in the job market season (e.g., March or later) may be of shorter duration. In some circumstances, employers are under heavy pressure to give less time to candidates for various reasons. If that is absolutely necessary, we recommend that employers give candidates a minimum of one week to consider the offer, and that candidates be given advance notice of this (e.g., at the flyout stage) whenever possible. Rationale: Recently, there is concern about a rise in “exploding offers” – i.e., offers for which candidates are given too few days to sufficiently consider the offer and their alternatives. This can prevent candidates from learning about their options or comparing offers, and at the extreme can be coercive. Giving candidates two weeks (or, late in the job market season, at least one week) to consider an offer is a reasonable standard. We also ask that all employers indicate on EconTrack when they have completed or closed their search. Job market institutions and mechanisms Please keep in mind the various job market institutions and mechanisms created by the AEA to improve the job market: The JOE Network includes a database of job openings for economists. Employers may sign up here: https://www.aeaweb.org/joe/employer. Job candidates may search the database here: https://www.aeaweb.org/joe/listings. The JOE Network has an electronic clearinghouse for job candidates to submit job applications. Job candidates may register here: https://www.aeaweb.org/joe/candidate. The AEA Committee on the Job Market releases data and guidance on the job market here: https://www.aeaweb.org/joe/communications. EconTrack: a board on which employers can indicate when they have extended interview and flyout invitations, and closed their search: https://www.aeaweb.org/econtrack.   Thank you for helping to ensure a transparent and equitable job market for new Ph.D. economists.

Signaling for medical residencies: the first few years

Two papers report on signaling in Otolaryngology and Orthopaedic surgery.

 The Otolaryngology societies have a summary of the current state of affairs that's worth hearing. A number of specialties (including Oto) allow many signals, and these seem to be acting as a soft cap on applications, rather than as a signal of special interest as in specialties that (like Economics) allow only a small number of signals.

Preference Signaling in Otolaryngology—Past, Present, and Future: A Comment From the Society of University Otolaryngologists (SUO), Association of Academic Departments in Otolaryngology (AADO), and the Otolaryngology Program Directors Organization (OPDO)  by Steven D. Pletcher MD, Bradley F. Marple MD, David J. Brown MD, The Laryngoscope Early View,  First published: 04 July 2024  https://doi.org/10.1002/lary.31613

"The year 2020 was a year of change. The residency application process, already suffering from spiraling application numbers,1 now faced the COVID-19 pandemic with a loss of away rotations and apprehension about virtual interviews. In the face of change, the Otolaryngology Program Directors Organization Council (OPDO) approached the leadership of the Association of Academic Departments in Otolaryngology (AADO) and the Society of University Otolaryngologists (SUO) with a recommendation to implement preference signaling. This system, originally described in the economics PhD marketplace,2 allows students a set number of signals (Otolaryngology used 5 in its inaugural year) to send to programs of particular interest. 

...

"Following the lead of Otolaryngology, Urology, General Surgery, Internal Medicine, and Dermatology implemented preference signaling the following year. Since that time, signaling has grown exponentially and is now utilized in the residency application process of nearly every specialty. 

...

"In the 2024-2025 residency application cycle, the evolution of preference signaling continues. Building on Otolaryngology's experience, in the 2023 application cycle Orthopaedic Surgery implemented a high-signal approach, providing applicants with 30 signals. This transition shows promise for reversing the vexing problem of spiraling application numbers—“Big Signaling” has now been adopted by Otolaryngology and four additional specialties the majority of whom have shown a 25%–30% decrease in applications submitted per student saving students a combined $2.5 million in application fees alone. Obstetrics and Gynecology has piloted a tiered signaling system, providing three gold and 15 silver signals to their students. 

...

"Because the number of signals received by programs is not publicized, students are unable to reliably target programs where their signals are less likely to be diluted by competing signals. Specialties should consider providing voluntary “signal cohort” (i.e., my program received between 75 and 100 signals in the 2024 application cycle) data to help applicants make more informed signal decisions and programs with low signal numbers will likely attract additional candidates. 

...

"One of the key statistics to guide applicants in high signal specialties is the interview offer rate for non-signal applications: this helps define the value of applications beyond the set number of signals."

########

Preference Signaling in the Orthopaedic Surgery Match: Applicant and Residency Program Attitudes, Behaviors, and Outcomes, by Guthrie, Stuart Trent MD, FAOA1,a; Dagher, Tanios BSE2; Essey-Stapleton, Jodi MS, MEd3; Balach, Tessa MD, FAOA2,  JBJS Open Access 9(2):e23.00146, April-June 2024. | DOI: 10.2106/JBJS.OA.23.00146

"In the first year of preference signaling, applicants reported applying to 16% fewer programs than if preference signaling had not been available. These results align with AAMC data, which report applications per program dropping 17.4% (from an average of 639.6 to 774.6), and applications per applicant dropping to 76.9 from 86.07,8. Further changes could occur in future cycles as students become more accustomed to the influence of signaling on their application."

NRMP Board of Directors (2020-2024), and a 40th anniversary

 This bit of glass marks the end of my term on the National Resident Matching Program (NRMP) Board of Directors.

One of the issues that consumed a lot of attention during my term is discussed in this post:

Friday, April 21, 2023

Transition from medical school to residency: defending the parts that work well (namely the NRMP Resident Match)

And here are all my posts about residents and fellows, going back to the beginning of this blog in 2008. (It's been interesting watching medical specialties begin to develop signaling in ways  reminiscent of signaling in the Economics job market, to deal with congestion of interviews and applications.*)

My first paper dealing explicitly with The Match suddenly seems to have been published 40 years ago:

Roth, A.E. "The Evolution of the Labor Market for Medical Interns and Residents: A Case Study in Game Theory", Journal of Political Economy, Vol. 92, 1984, 991‑1016. http://web.stanford.edu/~alroth/papers/evolut.pdf 

And the main report (with Elliott Peranson) of our redesign of The Match is now a quarter of a century old:

Roth, A.E. and E. Peranson, "The Redesign of the Matching Market for American Physicians: Some Engineering Aspects of Economic Design,” American Economic Review, 89, 4, September, 1999, 748-780. https://www.aeaweb.org/articles?id=10.1257/aer.89.4.748

*See yesterday's post for some discussion of market design interventions in job markets.

Social Media and Job Market Success: A Field Experiment on Twitter, by Qiu, Chen, Cohn, and Roth

 Here's a new working paper on SSRN:

Social Media and Job Market Success: A Field Experiment on Twitter, by Jingyi Qiu, Yan Chen, Alain Cohn, and Alvin E. Roth, May 20, 2024

"Abstract: We conducted a field experiment on Twitter to examine the impact of social media promotion on job market outcomes in economics. Half of the 519 job market papers tweeted from our research account were randomly assigned to be quote-tweeted by prominent economists. Papers assigned to be quote-tweeted received 442% more views and 303% more likes. Moreover, candidates in the treatment group received one additional flyout, with women receiving 0.9 more job offers. These findings suggest that social media promotion can improve the visibility and success of job market candidates, especially for underrepresented groups in economics such as women."

I gather that our paper has gone somewhat viral on twitter, with discussion about whether field experiments on job markets are ethical.  That's not a bad discussion to have, and of course we discussed that in the course of planning this experiment. (A similar discussion can and should be be had about any intervention in a market, not just an experiment.*)

Here is what we had to say about that in the paper.

"Despite the positive outcomes, one might question the ethics of our intervention, which randomly promotes a subset of JMPs on social media. However, we observe that senior economists naturally promote their own students and coauthors on Twitter. In comparison, we tweeted every JMP in our sample from our dedicated research account. Furthermore, while 80% of the influencers in our sample come from top 30 institutions, they quote-tweeted JMPs from a broader spectrum of academic institutions, thus allocating attention more equitably. Given that 92% of the JMCs in both the treatment and control groups accepted a job, it is unlikely that our treatment displaced those in the control group. The current focus on diversity, equity, and inclusion suggests that highlighting suitable candidates could potentially expand the number of job openings, making the job market for economists not entirely zero-sum. Lastly, the differential benefit of our treatment for women contributes to fostering a more inclusive economics profession. In summary, we argue that the knowledge gained from our experiment outweighs the potential cost."

*I've been involved in several operational (i.e. not experimental) interventions in job markets, including  the job market for new Econ Ph.D.s (e.g. signaling and the scramble):  see 

 Coles, Peter, John H. Cawley, Phillip B. Levine, Muriel Niederle, Alvin E. Roth, and John J. Siegfried, “The Job Market for New Economists: A Market Design Perspective,” Journal of Economic Perspectives, 24,4, Fall 2010, 187-206.  https://www.aeaweb.org/articles?id=10.1257/jep.24.4.187

Signaling in medical residency applications

 We're starting to see descriptive studies of how signals are being used in the labor market for new doctors.  Each medical specialty has chosen to adapt the kinds of signals used in Economics in its own way, with some specialties using only a handful of signals and others eliciting as many as 30.

Here are two papers from a recent issue of Academic Medicine.

Impact of Applicants’ Characteristics and Geographic Connections to Residency Programs on Preference Signaling Outcomes in the Match, by Benjamin, William J. MPH; Lenze, Nicholas R. MD, MPH; Bohm, Lauren A. MD; Thorne, Marc C. MD, MPH; Abraham, Reeni MD; Sepdham, Dan MD; Mihalic, Angela P. MD; Kupfer, Robbi A. MD,  Academic Medicine 99(4):p 437-444, April 2024. | DOI: 10.1097/ACM.0000000000005551

"Abstract

Purpose : To assess the impact of applicant and residency program characteristics on preference signaling outcomes in the Match during the first 2 years of implementation across 6 specialties.

Method : Data were obtained from the Texas Seeking Transparency in Application to Residency survey for applicants applying into otolaryngology during the 2020–2021 and 2021–2022 application cycles and into dermatology, internal medicine (categorical and preliminary year), general surgery, and urology during the 2021–2022 application cycle. The primary outcome was signal yield, defined as the number of interviews at signaled programs divided by the total number of signals sent. Associations with applicant-reported characteristics and geographic connections to residency programs were assessed using Wilcoxon rank sum testing, Spearman’s rank correlation testing, and ordinary least squares regression.

Results : 1,749 applicants with preference signaling data were included from internal medicine (n = 884), general surgery (n = 291), otolaryngology (n = 217), dermatology (n = 147), urology (n = 124), and internal medicine preliminary year (n = 86). On average 60.9% (standard deviation 32.3%) of signals resulted in an interview (signal yield). There was a stepwise increase in signal yield with the percentage of signals sent to programs with a geographic connection (57.3% for no signals vs. 68.9% for 5 signals, P < .01). Signal yield was positively associated with applicant characteristics, such as United States Medical Licensing Exam Step 1 and 2 scores, honors society membership, and number of publications (P < .01). Applicants reporting a lower class rank quartile were significantly more likely to have a higher percentage of their interviews come from signaled programs (P < .01).

Conclusions: Signal yield is significantly associated with geographic connections to residency programs and applicant competitiveness based on traditional metrics. These findings can inform applicants, programs, and specialties as preference signaling grows."

And here are the introductory paragraphs:

"The rising number of residency applications submitted per applicant has led to concerns that programs will not be able to adequately perform a holistic review of all applications and will instead rely on easily reviewed metrics, such as United States Medical Licensing Exam (USMLE) scores, class rank, and medical school reputation.1,2 In addition, COVID-19–related changes to the residency application process, such as the introduction of virtual interviewing and a cap on the number of away rotations medical students can complete, have limited applicants’ ability to informally express their interest in programs.3 Further, there is evidence that, while applying to the maximum number of programs is advantageous at the individual level, it leads to poorer overall results when all applicants follow this practice.2 To address this issue, new systems have been proposed, including personalized application paragraphs,4 program-specific messages,5 and preference signals.4,6,7

"Otolaryngology implemented a preference signaling system in 2021, which was based on theory developed by the American Economic Association (AEA) in 2006. The AEA used a preference signaling system for job market applicants, whereby applicants were allowed to express special interest in particular employers in their applications.8 Results from the AEA program highlighted that preference signals were beneficial to both candidates and employers in a labor market where employers are unable to provide full attention to every application they receive.9 Building off previous economic work, a computer simulation study run on 2014 otolaryngology Match data found that the number of interview invitations improved when applicants provided preferences on their Electronic Residency Application Service application; this result would have benefitted both programs and applicants.2

"Based on this research, the Otolaryngology Program Directors Association formally implemented a preference signaling system during the 2020–2021 application cycle in which applicants were granted 5 “signals” to send to residency programs prior to interviews indicating their strong interest in that program. Each program then received a list of the applicants who had sent them a signal.6,10,11 Data from the 2021 otolaryngology Match were notable for significantly increased interview rates at signaled programs across all levels of applicant competitiveness.10,12 Furthermore, the majority of program directors and applicants strongly supported the continuation of preference signaling.10,11 During the 2021–2022 application cycle, preference signaling pilot programs were implemented in 5 additional specialties: dermatology, internal medicine (categorical), internal medicine preliminary year, surgery (categorical), and urology, with each specialty using 5 signals per applicant, except dermatology, which used 3 signals"

#########

The Relationship Between Program and Applicant Characteristics With Applicant Program Signals in the 2022 Residency Recruitment Cycle: Findings From 3 Specialties, by LaFemina, Jennifer MD; Rosman, Ilana S. MD; Wallach, Sara L. MD; Wise, Paul E. MD; Smink, Douglas S. MD, MPH; Fletcher, Laura PhD, Academic Medicine 99(4):p 430-436, April 2024. | DOI: 10.1097/ACM.0000000000005586

"Abstract

Purpose: Continuing increases in application volume have driven a national dialogue to reform the residency recruitment process. Program signaling allows applicants to express interest in a program at the preinterview stage with the goal of helping programs identify applicants with more genuine interest in their programs. This study explored the relationship between program signals and program and applicant characteristics.

Method: Participating dermatology, general surgery, and categorical internal medicine (IM) programs and applicants of the 2022 supplemental ERAS application (SuppApp) were included. Data from the SuppApp, the MyERAS Application for Residency Applicants (MyERAS), and the 2020 GME Track Survey were used. Cohen’s h was used to determine effect size, and chi-squared was used to determine statistical significance.

Results:There was an uneven distribution of signals to programs, with 25% of programs receiving about half of the signals across all 3 specialties. Programs with larger numbers of both residents and applicants received greater numbers of program signals relative to their program density, although this effect was small (h < 0.50, P < .001). No meaningful differences were seen across genders for any specialty. Only Hispanic applicants in IM sent a higher proportion of signals to programs with more underrepresented in medicine residents than White only applicants (40% vs 26%, h = 0.30, P < .001). Across all specialties, there was a small-to-moderate effect for international medical graduate (IMG) applicants sending a larger proportion of signals to programs with more IMG residents (h < 0.80, P < .001).

Conclusions: This first-year pilot study (i.e., SuppApp) provided initial evidence that supports the feasibility and fairness of program signals in residency selection. As program signals become more common across specialties, future research should continue to evaluate trends in where applicants send signals, and possible relationships between program and application characteristics."

"IMG applicants were more likely to signal programs with a greater proportion of IMG residents. The effect was small in dermatology and increased to moderate in GS and large in IM. In the NRMP’s 2022 Main Residency Match, 11 IMGs (U.S. and non-U.S.) matched into postgraduate year 2 dermatology, representing 2% of positions. This compares to the 10% and 38% IMG Match rate into GS and IM, respectively.21 While at this time, correlation of signal distribution and the likelihood of successfully matching is not available, these findings suggest that in general, IMG applicants sent more signals to programs they knew to be “IMG friendly” (i.e., more likely to accept IMGs), which they could easily identify with tools such as the Residency Explorer Tool22 and the Residency Programs List.23 However, if IMGs continue to send more signals to programs with already higher proportions of IMG residents, this may maintain the status quo or even further restrict the IMG applicant pool all programs are willing to consider during their resident selection process because programs with fewer IMGs will continue to receive a lower proportion of signals from IMG applicants. This could ultimately negatively affect diversity across programs"

Signaling for residency programs in dermatology, general surgery, and internal medicine

We're starting to see some data from signaling for residency applications.  This paper observes that programs are more likely to interview candidates who send them a signal. (Economists will worry that this reflects which programs are signaled and not just the effect of a signal...)  These three specialties have relatively few signals, more like economics than like Orthopedic Surgery (which has 30 signals).  And the table indicates that more interviews are offered than signals received, so that's another difference from Ortho...)

Rosenblatt, Adena E., Jennifer LaFemina, Lonika Sood, Jennifer Choi, Jennifer Serfin, Bobby Naemi, and Dana Dunleavy. "Impact of Preference Signals on Interview Selection Across Multiple Residency Specialties and Programs." Journal of Graduate Medical Education 15, no. 6 (2023): 702.

"Abstract

"Background Program signaling is an innovation that allows applicants to express interest in specific programs while providing programs the opportunity to review genuinely interested applicants during the interview selection process.

"Objective To examine the influence of program signaling on “selected to interview” status across specialties in the 2022 Electronic Residency Application Service (ERAS) application cycle.

"Methods Dermatology, general surgery-categorical (GS), and internal medicine-categorical (IM-C) programs that participated in the signaling section of the 2022 supplemental ERAS application (SuppApp) were included. Applicant signal data was collected from SuppApp, applicant self-reported characteristics collected from the MyERAS Application for Residency Applicants, and 2020 program characteristics collected from the 2020 GME Track Survey. Applicant probability of being selected for interview was analyzed using logistic regression, determined by the selected to interview status in the ERAS Program Director’s WorkStation.

"Results Dermatology had a 62% participation rate (73 of 117 programs), GS a 75% participation rate (174 of 232 programs), and IM-C an 86% participation rate (309 of 361 programs). In all 3 specialties examined, on average, signaling increased the likelihood of being selected to interview compared to applicants who did not signal. This finding held across gender and underrepresented in medicine (UIM) groups in all 3 specialties, across applicant types (MDs, DOs, international medical graduates) for GS and IM-C, and after controlling for United States Medical Licensing Examination Step 1 scores.

"Conclusions Although there was variability by program, signaling increased likelihood of being selected for interview without negatively affecting any specific gender or UIM group."

Data from future years will be needed to determine how signaling is influencing the distribution of residents to programs.

Applying for medical residencies: a consensus statement from Internal Medicine

 The Alliance for Academic Internal Medicine has released a "consensus statement" with many proposals about application and interview caps, and signaling.

Catalanotti, Jillian S., Reeni Abraham, John H. Choe, Kelli A. Corning, Laurel Fick, Kathleen M. Finn, Stacy Higgins et al. "Rethinking the Internal Medicine Residency Application Process to Prioritize the Public Good: A Consensus Statement of the Alliance for Academic Internal Medicine." The American Journal of Medicine (2023).

It also includes a call for data and analysis:

"AAIM proposes increasing internal medicine program preference signals to 15, using tiered signaling with three “gold” and 12 “silver” signals, and setting an interview cap of 15 in the 2024-2025 recruitment season, with participation by all internal medicine programs. The Alliance recommends that all internal medicine programs participate in ACI. AAIM recommends that programs transparently share information about their use of preference signals and other application screening methods and calls for real-time data analysis to explore impact, inform future iterations and identify potential harms.

"The Alliance calls upon ERAS and NRMP as well as Thalamus® and other interview scheduling platforms to transparently share data, to embrace change, and to perform analyses needed to inform this process. For example, recent modeling with eight years of retrospective NRMP data in OBGYN demonstrated that an early match round may increase the number of “mutually dissatisfied applicant-program pairs” and that a multiple-round match process could introduce potential rewards for gamesmanship, a prime factor addressed by the current process.35 AAIM applauds this analysis and hopes that the new collaboration between ERAS and Thalamus® may provide useful interview data to inform this proposal and further interventions."

And here is reference 35 in that last paragraph, about which I've blogged before.

I Ashlagi, E Love, JI Reminick, AE. Roth

Early vs Single Match in the Transition to Residency: Analysis Using NRMP Data From 2014 to 2021

J Grad Med Educ, 15 (2) (Apr 2023), pp. 219-227, 10.4300/JGME-D-22-00177.1

View in ScopusGoogle Scholar

Signaling for Orthopaedic surgery residencies

 The Journal of Bone and Joint Surgery reports a small survey about signaling behavior among applicants for orthopaedic surgery residencies.

Deckey, David G., Eugenia Lin, Coltin RB Gerhart, Joseph C. Brinkman, Karan A. Patel, and Joshua S. Bingham. "Decoding the Signals: An Analysis of Preference Signaling in the 2023 Orthopaedic Surgery Residency Match." JBJS Open Access 8, no. 3 (2023).

"While previously used in other specialties, the preference signaling program (PSP) was implemented in the 2022 to 2023 orthopaedic surgery residency application process for the first time. The PSP allowed for 30 signaling tokens to be sent by applicants to programs of their choice to indicate particular interest in a program.

...

"An anonymous electronic survey was emailed to all orthopaedic surgery residency applicants who applied to the authors' institution during the 2022 to 2023 application cycle. The survey was sent after match lists were submitted and closed before the release of match results. 

...

"The survey was completed by 101 applicants. Applicants applied to a mean of 90 programs (range: 10-197) and received an average of 12 interview invitations (range: 0-39). Applicants almost uniformly used all 30 signals, with nearly two-thirds signaling their home programs (65%, 49/76), and nearly all applicants sending signals to programs at which they performed away rotations (95.7%, 88/92). Applicants received a mean of 9 invitations from programs they signaled, compared with 2 invitations from programs they did not signal."

#######

Applicants report sending signals to all the programs that would have been expected to automatically give them interviews even in the absence of a signaling mechanism--namely their home programs and those which they have spent time visiting in 'away rotations.'

In the Economics job market, which may have been the first to introduce signaling, we limited applicants to 2 signals, and advised them not to signal jobs in which they already had well established mutual interests, i.e. not to signal jobs which they felt would interview them without signals. One of the ideas behind the Econ signaling mechanism is that there are many ways applicants can send signals of interest within a network to which they are well connected. Since some applicants are better connected than others, we were offering some signals that could be sent out of network.

It will be interesting to understand if signals to Ortho residencies are helping promote out of network interviews, or are largely strengthening the network connections already established by medical schools (home programs) and away rotations.

OB-GYN doctors will use a new application system to apply to residency programs (but will continue to go through the NRMP resident match)

 Before new doctors can participate in the resident match (by engaging with the NRMP), they first have to apply to residency programs, and arrange interviews.  This process has been experiencing congestion, and the specialty of Obstetrics and Gynecology has now decided to switch application services. 

However, participation in the NRMP will not change: the Association of Professors of Gynecology and Obstetrics (APGO) FAQ states "Obstetrics and gynecology applicants will use the National Residency Match Program (NRMP) for the Match. This new application does not change how the applicant or programs interact with the NRMP Match system."

Medpage Today has the story:

Ob/Gyn Switching to Independent System for Residency Applications— This is the last year ob/gyn will use ERAS, despite helping to pilot the program  by Rachael Robertson, Enterprise & Investigative Writer, MedPage Today August 25, 2023

"Beginning next year, ob/gyn programs will start using an independent system for processing residency applications, rather than the Electronic Residency Application Service (ERAS).

"The joint decision to switch to the new system was made by the American College of Obstetricians and Gynecologists (ACOG), the Association of Professors of Gynecology and Obstetrics (APGO), and the Council on Resident Education in Obstetrics and Gynecology (CREOG). The new system will be managed by Liaison International, which uses "Centralized Application Service (CAS) technology," according to the company's website.

...

"A joint statement on the APGO website opens in a new tab or window

said that the new system "will be user friendly and efficient, less expensive for applicants, and will directly decrease the burdens faced by program directors, program managers, and applicants alike," and "will incorporate the entirety of interview season functions, from application submission, review, interview offers and interviews, to rank list submission."

ACOG explained that the decision to pull the ERAS stemmed from the Right Resident, Right Program, Ready Day One initiativeopens in a new tab or window, noting that the new system is mobile-friendly and "will include immediate fee reduction," as detailed on their FAQ pageopens in a new tab or window.

In response, the Association of American Medical Colleges (AAMC), which runs ERAS, issued a statementopens in a new tab or window attributed to President and CEO David J. Skorton, MD, and Alison J. Whelan, MD, the chief academic officer, saying they were "surprised and dismayed" by the decision. 

...

"Bryan Carmody, MD, of Eastern Virginia Medical School in Norfolk, shared information  opens in a new tab or window

about the change on social media, writing on his blogopens in a new tab or window that ob/gyn program directors helped to pilot ERAS when it was first rolled out in the mid-90s.

Carmody told MedPage Today that he anticipates the biggest downsides will fall on applicants, such as those who want to apply to another specialty in addition to ob/gyn.

"Those applicants will have to use one system to apply to ob/gyn and another to their other specialty," he explained. "The same thing applies to applicants who fail to match. They'll have to use ERAS to apply to another specialty during SOAP [Supplemental Offer and Acceptance Program] since few, if any, ob/gyn positions are typically available."

*************

OB-GYN will continue to employ signaling  (very loosely modeled on the signaling used in the Econ PhD job market, but asking applicants to submit 3 "gold" signals and 15 "silver" signals): 

Program Signaling for OBGYN Residency Application Background and FAQs

**************

Earlier:

Friday, April 21, 2023

Transition from medical school to residency: defending the parts that work well (namely the NRMP Resident Match)

Transition from medical school to residency: defending the parts that work well (namely the NRMP Resident Match)

This post is about a recently published paper concerning the design of the market for new doctors in the U.S.  But it will require some background for most readers of this blog.   The short summary is that the market is experiencing problems related to congestion, and one of the proposals to address these problems was deeply flawed, and would have reduced market thickness and caused substantial direct harm to participants if implemented, and created instabilities that would likely have caused indirect harms to the match process in subsequent years. But this needed to be explained in the medical community, since that proposal was being  very actively advocated.

For those of you already steeped in the background, you can go straight to the paper, here.

Itai Ashlagi, Ephy Love, Jason I. Reminick, Alvin E. Roth; Early vs Single Match in the Transition to Residency: Analysis Using NRMP Data From 2014 to 2021. J Grad Med Educ 1 April 2023; 15 (2): 219–227. doi: https://doi.org/10.4300/JGME-D-22-00177.1

If the title doesn't remind you of the vigorous advocacy for an early match for select positions, here is some of the relevant back story.

The market for new doctors--i.e. the transition from medical school to residency--is experiencing growing pains as the number of applications and interviews has grown, which imposes costs on both applicants and residency programs.  

Below is a schematic of that process, which begins with applicants submitting applications electronically, which makes it easy to submit many.  This is followed by residency programs inviting some of their applicants to interview. The movement to Zoom interviews has made it easier to have many interviews also (although interviews were multiplying even before they moved to Zoom).  

After interviews, programs and applicants participate in the famous centralized clearinghouse called The Match, run by the NRMP. Programs and applicants each submit rank order lists (ROLs) ranking those with whom they interviewed, and a deferred acceptance algorithm (the Roth-Peranson algorithm) produces a stable matching, which is publicly announced on Match Day. (Unmatched people and positions are invited into a now computer-mediated scramble, called SOAP, and these matches too are announced on Match  Day.)

The Match had its origins as a way to control the "unraveling" of the market into inefficient bilateral contracts, in which employment contracts were made long before employment would commence, via exploding offers that left most applicants with very little ability to compare options.  This kind of market failure afflicted not only the market for new physicians (residents), but also the market for later specialization (as fellows). Consequently, over the years, many specialties have turned to matching for their fellowship positions as well.

  The boxes in brown in the schematic are those that constitute "The Match:" the formulation and submission of the ROLs, and the processing of these into a stable matching of programs to residents.  Congestion is bedeviling the parts in blue.

The boxes colored brown are 'The Match' in which participants formulate and submit rank order lists (ROLs), after which a deferred acceptance algorithm produces a stable matching of applicants to programs, which is accepted by programs and applicants on Match Day. The boxes in blue, the applications and interviews that precede the Match, are presently suffering from some congestion.  Some specialties have been experimenting with signals (loosely modeled on those in the market for new Economics PhDs, but implemented differently by different medical specialties).

The proposal in question was to divide the match into two matches, run sequentially, with the first match only allowing half of the available positions to be filled.  The particular proposal was to do this first for the OB-GYN specialty, thus separating that from the other specialties in an early match, with only half of the OB-GYN positions available early.

This proposal came out of a study funded by the American Medical Association, and it was claimed, without any evidence being offered, that it would solve the current problems facing the transition to residency.  Our paper was written to provide some evidence of the likely effects, by simulating the proposed process using the preferences (ROLs) submitted in previous years.  

The results show that the proposal would largely harm OB-GYN applicants by giving them less preferred positions than they could get in a traditional single match, and that it would create instabilities that would encourage strategic behavior that would likely undermine the successful operation of the match in subsequent years.

Itai Ashlagi, Ephy Love, Jason I. Reminick, Alvin E. Roth; Early vs Single Match in the Transition to Residency: Analysis Using NRMP Data From 2014 to 2021. J Grad Med Educ 1 April 2023; 15 (2): 219–227. doi: https://doi.org/10.4300/JGME-D-22-00177.1

Abstract:

"Background--An Early Result Acceptance Program (ERAP) has been proposed for obstetrics and gynecology (OB/GYN) to address challenges in the transition to residency. However, there are no available data-driven analyses on the effects of ERAP on the residency transition.

"Objective--We used National Resident Matching Program (NRMP) data to simulate the outcomes of ERAP and compare those to what occurred in the Match historically.

"Methods--We simulated ERAP outcomes in OB/GYN, using the de-identified applicant and program rank order lists from 2014 to 2021, and compared them to the actual NRMP Match outcomes. We report outcomes and sensitivity analyses and consider likely behavioral adaptations.

"Results--Fourteen percent of applicants receive a less preferred match under ERAP, while only 8% of applicants receive a more preferred match. Less preferred matches disproportionately affect DOs and international medical graduates (IMGs) compared to US MD seniors. Forty-one percent of programs fill with more preferred sets of applicants, while 24% fill with less preferred sets of applicants. Twelve percent of applicants and 52% of programs are in mutually dissatisfied applicant-program pairs (a pair in which both prefer each other to the match each received). Seventy percent of applicants who receive less preferred matches are part of a mutually dissatisfied pair. In 75% of programs with more preferred outcomes, at least one assigned applicant is part of a mutually dissatisfied pair.

"Conclusions--In this simulation, ERAP fills most OB/GYN positions, but many applicants and programs receive less preferred matches, and disparities increase for DOs and IMGs. ERAP creates mutually dissatisfied applicant-program pairs and problems for mixed-specialty couples, which provides incentives for gamesmanship."

************

I'm hopeful this paper will effectively contribute to the ongoing discussion of how, and how not, to modify the design of the whole process of transition to residency with an aim to fixing the parts that need fixing, without damaging the parts that work well, i.e. while doing no harm. 

(Signaling will likely continue to play a role in this.)