AI assisted job applications (and application screening)

 NBC news has the story:

AI is supposed to make applying to jobs easier — but it might be creating another problem
Companies have touted new AI technology that allows users to apply to thousands of jobs per day, flooding job openings with resumes.  By Sophia Pargas

"With the help of new AI software, applicants can now revise their resumes and cover letters, receive live interview scripts and launch chatbots to submit thousands of applications almost instantly.

...

"LazyApply, Simplify and AI Hawk are all AI services that launch assistants to help collect applicant information and submit automated job applications through sites like LinkedIn and Indeed. Using the tools, job seekers can save hundreds of hours — and apply to up to thousands of jobs a day, according to the services.

"AI Apply, which claims its users are “80% more likely to get hired,” offers tools like a cover letter and resume builder, an auto-apply feature, an interview practice generator and a specialized interview buddy. A premium membership starts at $38 a month, according to the website — a cost AI Apply notes is much lower and more accessible than the cost of a career counselor.

...

"According to the company’s respective websites, OfferGoose has helped applicants land over 17,000 job offers, and AI Apply has over 340,000 “freelancers and job seekers” using the service.

"Nevertheless, many companies have enlisted additional safeguards to detect the use of AI. Some employers now require e-confirmation codes to submit applications and prevent automated submissions. Others have added prompts asking AI bots to use specific words like “banana” in responses to catch chat-generated application answers, according to longtime job recruiter and tech agency director of talent Mike Peditto.

"But job seekers are not the only party using AI in the application process, according to a recent University of Washington study. The researchers estimated that “99% of Fortune 500 companies are already using some sort of AI assistance when making hiring decisions,” and found that resume-screening tools have an incredible bias toward white, male applicants — with the large language models favoring white-associated males roughly 85% of the time and disadvantaging Black males in “up to 100% of cases.”

"The use of AI by both applicants and employers has created a distinct dynamic, in which applicants are using AI, but are also reticent about employers screening them with AI."

Many preference signals as a soft cap on number of applications in medical residency matching

 Here's a review article on matching for medical residents,  with particular attention to neurosurgery, in the Cureus Journal of Medical Science.  In specialties that (like neurosurgery) allow applicants to send many signals, many applicants signal to and match with programs with which they have some prior connection.

Ozair, Ahmad, Jacob T. Hanson, Donald K. Detchou, Matthew P. Blackwell, Abigail Jenkins, Marianne I. Tissot, Umaru Barrie et al. "Program Signaling and Geographic Preferences in the United States Residency Match for Neurosurgery." Cureus 16, no. 9 (2024).

Abstract: Postgraduate residency training has long been the cornerstone of academic medicine in the United States. The Electronic Residency Application Service (ERAS), managed by the Association of American Medical Colleges (AAMC), is the central residency application platform in the United States for most clinical specialties, with the National Residency Matching Program (NRMP) being the algorithm for matching residency programs with applicants. However, the determination of the best fit between ERAS applicants and programs has been increasingly challenged by the rising number of applicants per residency spot. This application overburdening across competitive specialties led to several adverse downstream effects, which affected all stakeholders. While several changes and proposals were made to rectify the issue of application overburdening, the 2020-2021 ERAS Match Cycle finally saw several competitive specialties, including otolaryngology and urology, utilize a new system of supplemental residency application based on preference signals/tokens. These tokens permit applicants to electronically signal a select number of programs in a specialty of choice, with the program reviewing the application now cognizant that they have been signaled, i.e., the applicant has chosen to use up a limited set of signals for their program. Initial results from otolaryngology and urology, as described in this article, indicated the value of this new system to both applicants and educators. Given the favorable outcomes and broader uptake of the system among other specialties, the field of neurosurgery adopted the utilization of the ERAS-based program signaling and geographic preference for the first time for the 2022-2023 Residency Application Cycle and later opted to continue them for the 2023-2024 and 2024-2025 cycles. For the 2024-2025 Match Cycle, neurosurgery applicants have 25 signals, i.e., a "high-signal" approach, where non-signaled programs have a low interview conversion rate. This literature review discusses the rationale behind the change, the outcomes of other competitive specialties from prior cycles, the evolving nature of the change, and the potential impact on applicants and programs. As we describe in this review, signaling may potentially represent a surrogate form of an application cap. Other considerations relate to cost savings for both applicants and programs from a high-signal approach in neurosurgery. These modifications represent a foundational attempt to alleviate the application overburdening and non-holistic review in the residency application process, including for neurosurgery. While these changes have been a welcomed addition for all stakeholders in residency match cycles so far, further prospectively directed surveys along with qualitative research studies are warranted to better delineate the downstream impact of these changes and guide further optimization of the application system.

Preference signaling for urologists (including Flush Day)

 It's been very interesting for me to watch the growth of preference signaling to combat congestion in applications and interviews, from its beginnings in the job market for new Ph.D. economists, to its spread through the medical specialty job markets hosted by the National Resident Matching Program (NRMP), and on to medical specialties that run their own matches, such as urology.

So, if you want to be a urologist, take note of the signaling deadline today:

The Society of Academic Urologists Resident Match Program 

Applicant Preference Signaling Deadline is today, September 16, 2024

The SAU has a good deal of supporting material on their site. 

Here's a summary of their interview and offer procedures: Interview Offer Summary (which includes an appropriately named Flush Day for finalizing interviews).

And here's their page on Preference Signaling, linking to this Overview, FAQs, and webinar.

The SAU invites applicants to send up to 30 signals, and encourages them to include programs that know them well on their signaling list.  So (as I've remarked in previous posts) I think this is likely to work as a soft cap on applications.

HT: Mike Rees

Preference signaling in the Political Science job market

 There is now signaling in the Political Science job market:  They allow job candidates to send up to three signals: here's the APSA's page with frequently asked questions about signaling.

Aside from the procedures APSA members should use, their site offers some general comments about signaling. Some excerpts:

What is the history of signaling as applied to specialized labor markets like that for political science PhDs?

 The American Economics Association (AEA) Ad Hoc Committee on the Job Market, including Roth, applied his research to the economics PhDs job market over 15 years ago. Since the 1970s, the job market for economists has been organized around the AEA and related associations’ annual meeting (called the Allied Social Sciences Associations, or the “ASSA”) in January and since 2006, economists have utilized signaling for job interviews that take place at the ASSA. For those who don’t secure a position there, a job “scramble” that is more public takes place later in the spring. These events act as clearinghouses to clear congestion resulting from candidates applying for almost all positions due to their need to secure a position, and also prevents “unraveling,” where employers make offers earlier and earlier each year to get the best possible candidate they can.[1] The AEA Ad Hoc Committee on the Job Market offers detailed advice on signaling for (economics) PhDs going on the job market here.

Why would the political science job market benefit from signaling?

The number of political science job postings per PhD on the job market has decreased considerably since 2011, when there were two job openings for each candidate as it recovered from the Great Recession.[1] After the COVID pandemic, the higher education job market seems to be showing signs that it is recovering more slowly, and political science is no exception to this trend. The behavior of the higher education job markets runs counter to the general labor market trend in the US where jobs are currently outnumbering jobseekers.[2]  Currently there are almost two candidates per job opening on APSA eJobs.[3] The political science job market is also congested in 2023. In the last few years, candidates have reported applying for upwards of 25-50, more than 51, or more than 100 positions in a job market cycle, making the possibilities for matches endless.[4]

After the job market at APSA’s Annual Meeting in September, the job market for political science PhDs is decentralized and thin, lacking a centralizing timeframe, location, or deadline, with the only way for candidates to let employers know they are interested is to contact the hiring committee or send in an application for the position. Jobs are being posted earlier each year to secure the best candidates by utilizing the APSA Annual Meeting as a job market clearinghouse. For the remainder of the academic year, those participating in the job market lack any kind of intentionally designed mechanism to structure their path to a job placement before the next academic year..[5] For all these reasons: a surplus of jobseekers, market congestion (applying to all jobs), market unraveling (earlier and earlier offers), the political science job market would benefit from being approached with the market design tools that have been shown to alleviate these inefficiencies.

Citations

How will signaling benefit candidates and employers at APSA Annual Meeting/ Interview Services?

 Due to job market congestion, the job market centralized around the APSA Annual Meeting/ Interview Services would benefit from a mechanism for candidates and employers to be able to decipher meaningful interest more easily from candidates for positions, and to allow institutions to view candidates more equally across characteristics outside of institutional prestige, rank, or publication records. Due to the large number of candidates relative to jobs, we believe signaling can increase the number of matches facilitated, decrease the number of applications both written and submitted by candidates and reviewed by hiring committees, and decrease the stress of all participants in the market. We have created a pilot program to beta test how signaling will work for employers and candidates utilizing Interview Services at the Annual Meeting.

Interviews, flyouts and offers in the job market for new PhD economists, 2024-25

 We may still be in a period of disequilibrium in the conduct of the market for new economists, as Zoom interviews have replaced in-person interviews at the annual January ASSA meetings.  In an attempt at maintaining market thickness, the AEA has reaffirmed its guidelines, as discussed in the email below that went out to AEA members yesterday

AEA Guidance on the 2024-25 Economics Job Market Cycle

Date: August 5, 2024 To: Members of the American Economic Association From: Peter L. Rousseau, Secretary-Treasurer The AEA Executive Committee, in conjunction with its Committee on the Job Market, recognizes that it is to the benefit of the profession if the job market for economists is thick, with many employers and job candidates participating in the same stages at the same time. Moreover, the AEA's goals of diversity, equity, and inclusion are fostered by having a timeline that remains widely known and accepted, ensuring that candidates can correctly anticipate when each stage will occur. The AEA has a role to establish professional norms, which includes ensuring fair treatment of job candidates, including that they have enough time to consider job offers. With these goals in mind, and in light of inquiries from both job candidates and employers about how to proceed, the AEA asks that departments and other employers consider the following timeline for initial interviews and “flyouts” in the upcoming job cycle (2024-25). Timing of interview invitations The AEA suggests that employers wait to extend interview invitations until the day after job market signals are transmitted to employers (planned for December 4). Rationale: the AEA created the signaling mechanism to reduce the problem of asymmetric information and allow job candidates to credibly signal their interest to two employers. The AEA asks that employers wait to extend interview invitations until those signals have been transmitted, and to use that information to finalize their set of candidates to interview. This helps the job market in several ways: it reduces the problem of imperfect information, it helps ensure a thick market at each stage, and it promotes the AEA’s goals of diversity, equity, and inclusion. Job candidates from historically under-represented groups may lack informal professional networks and thus may especially rely on the signals to convey their interest. Waiting to review the signals before issuing invitations promotes a fairer, more equitable process. We also ask that all employers indicate on EconTrack when they have extended interview invitations. This allows candidates to learn about the status of searches without visiting websites posting crowd-sourced information and potentially inappropriate other content. Timing of interviews Initial interviews may take place any time after the AEA signals are sent (planned for December 4). The AEA recommends that all initial interviews take place virtually (e.g., by Zoom). We suggest that interviews not take place during the ASSA meeting itself (January 3-5, 2025). Rationale: In the past, interviews were conducted in-person at the ASSA meetings. This promoted thickness of the market, because most candidates and employers were present at the meetings but had the disadvantage of precluding both job candidates and interviewers from fully participating in sessions. Initial job interviews went online during COVID, and feedback indicated that the benefits of virtual first-round interviews (e.g., low monetary cost, zero cost in travel time, scheduling flexibility, convenience) outweighed the limitations (e.g., less rich interaction). We ask that interviews NOT take place during the ASSA meetings (January 3-5, 2025) in order to allow job candidates and interviewers to participate in the conference. Timing of flyouts Flyouts have historically happened at times appropriate for the employer, and the AEA sees no reason to suggest otherwise. We ask that all employers indicate on EconTrack when they have extended flyout invitations. Unlike with interviews, the AEA does not take a position on whether flyouts should be virtual or in-person. Timing of offers and “exploding” offers In order to ensure that the job market remains sufficiently synchronized and thick, and that candidates have a chance to compare offers, the AEA recommends that employers leave job offers open (i.e. do not require candidates to accept or decline) until at least January 31. The AEA also strongly recommends that employers give candidates at least two weeks to consider their job offer. We recognize that offers made late in the job market season (e.g., March or later) may be of shorter duration. In some circumstances, employers are under heavy pressure to give less time to candidates for various reasons. If that is absolutely necessary, we recommend that employers give candidates a minimum of one week to consider the offer, and that candidates be given advance notice of this (e.g., at the flyout stage) whenever possible. Rationale: Recently, there is concern about a rise in “exploding offers” – i.e., offers for which candidates are given too few days to sufficiently consider the offer and their alternatives. This can prevent candidates from learning about their options or comparing offers, and at the extreme can be coercive. Giving candidates two weeks (or, late in the job market season, at least one week) to consider an offer is a reasonable standard. We also ask that all employers indicate on EconTrack when they have completed or closed their search. Job market institutions and mechanisms Please keep in mind the various job market institutions and mechanisms created by the AEA to improve the job market: The JOE Network includes a database of job openings for economists. Employers may sign up here: https://www.aeaweb.org/joe/employer. Job candidates may search the database here: https://www.aeaweb.org/joe/listings. The JOE Network has an electronic clearinghouse for job candidates to submit job applications. Job candidates may register here: https://www.aeaweb.org/joe/candidate. The AEA Committee on the Job Market releases data and guidance on the job market here: https://www.aeaweb.org/joe/communications. EconTrack: a board on which employers can indicate when they have extended interview and flyout invitations, and closed their search: https://www.aeaweb.org/econtrack.   Thank you for helping to ensure a transparent and equitable job market for new Ph.D. economists.

Signaling for medical residencies: the first few years

Two papers report on signaling in Otolaryngology and Orthopaedic surgery.

 The Otolaryngology societies have a summary of the current state of affairs that's worth hearing. A number of specialties (including Oto) allow many signals, and these seem to be acting as a soft cap on applications, rather than as a signal of special interest as in specialties that (like Economics) allow only a small number of signals.

Preference Signaling in Otolaryngology—Past, Present, and Future: A Comment From the Society of University Otolaryngologists (SUO), Association of Academic Departments in Otolaryngology (AADO), and the Otolaryngology Program Directors Organization (OPDO)  by Steven D. Pletcher MD, Bradley F. Marple MD, David J. Brown MD, The Laryngoscope Early View,  First published: 04 July 2024  https://doi.org/10.1002/lary.31613

"The year 2020 was a year of change. The residency application process, already suffering from spiraling application numbers,1 now faced the COVID-19 pandemic with a loss of away rotations and apprehension about virtual interviews. In the face of change, the Otolaryngology Program Directors Organization Council (OPDO) approached the leadership of the Association of Academic Departments in Otolaryngology (AADO) and the Society of University Otolaryngologists (SUO) with a recommendation to implement preference signaling. This system, originally described in the economics PhD marketplace,2 allows students a set number of signals (Otolaryngology used 5 in its inaugural year) to send to programs of particular interest. 

...

"Following the lead of Otolaryngology, Urology, General Surgery, Internal Medicine, and Dermatology implemented preference signaling the following year. Since that time, signaling has grown exponentially and is now utilized in the residency application process of nearly every specialty. 

...

"In the 2024-2025 residency application cycle, the evolution of preference signaling continues. Building on Otolaryngology's experience, in the 2023 application cycle Orthopaedic Surgery implemented a high-signal approach, providing applicants with 30 signals. This transition shows promise for reversing the vexing problem of spiraling application numbers—“Big Signaling” has now been adopted by Otolaryngology and four additional specialties the majority of whom have shown a 25%–30% decrease in applications submitted per student saving students a combined $2.5 million in application fees alone. Obstetrics and Gynecology has piloted a tiered signaling system, providing three gold and 15 silver signals to their students. 

...

"Because the number of signals received by programs is not publicized, students are unable to reliably target programs where their signals are less likely to be diluted by competing signals. Specialties should consider providing voluntary “signal cohort” (i.e., my program received between 75 and 100 signals in the 2024 application cycle) data to help applicants make more informed signal decisions and programs with low signal numbers will likely attract additional candidates. 

...

"One of the key statistics to guide applicants in high signal specialties is the interview offer rate for non-signal applications: this helps define the value of applications beyond the set number of signals."

########

Preference Signaling in the Orthopaedic Surgery Match: Applicant and Residency Program Attitudes, Behaviors, and Outcomes, by Guthrie, Stuart Trent MD, FAOA1,a; Dagher, Tanios BSE2; Essey-Stapleton, Jodi MS, MEd3; Balach, Tessa MD, FAOA2,  JBJS Open Access 9(2):e23.00146, April-June 2024. | DOI: 10.2106/JBJS.OA.23.00146

"In the first year of preference signaling, applicants reported applying to 16% fewer programs than if preference signaling had not been available. These results align with AAMC data, which report applications per program dropping 17.4% (from an average of 639.6 to 774.6), and applications per applicant dropping to 76.9 from 86.07,8. Further changes could occur in future cycles as students become more accustomed to the influence of signaling on their application."

Centralized assignment mechanisms that don't include all the relevant choices, by Kapor, Karnani and Neilson in the JPE

 One of the issues in organizing a centralized matching mechanism is to make the market thick, by including all or most of the relevant choices in the centralized system.  If not, there will be transactions outside of the centralized marketplace, and some of them may be costly to the system.

Here's a paper that explores that in the context of  college admissions in Chile.

Aftermarket Frictions and the Cost of Off-Platform Options in Centralized Assignment Mechanisms, by Adam Kapor, Mohit Karnani, and Christopher Neilson, Journal of Political Economy, forthcoming.

Abstract: "We study the welfare and human capital impacts of colleges’ (non)participation in Chile’s centralized higher-education platform, leveraging administrative data and two policy changes: the introduction of a large scholarship program and the inclusion of additional institutions, which raised the number of on-platform slots by approximately 40%. We first show that the expansion of the platform raised on-time graduation rates. We then develop and estimate a model of college applications, offers, wait lists, matriculation, and graduation. When the platform expands, welfare increases, and welfare, enrollment, and graduation rates are less sensitive to off-platform frictions. Gains are larger for students from lower-socioeconomic-status backgrounds."

"in virtually every practical implementation there exist many off-platform options that are available to participants of the match. In primary and secondary education, these include private schools or charter schools that do not participate in the centralized system. In other cases, such as higher education, some providers may be excluded from the platform by regulation, while others may choose not to participate. When off-platform options exist, applicants may renege on their assigned matches in favor of programs that did not participate in the centralized process. In turn, these decisions lead to the use of wait lists and aftermarkets, which may be inefficient due to the presence of congestion and matching frictions and can be inequitable if some students are better able to navigate this partially decentralized process, negating some of the benefits of the match.

"In this paper, we study the empirical relevance of the configuration of on- and off-platform options for students’ welfare and for persistence and graduation in higher-education programs. We document the importance of negative externalities generated by off-platform options and quantify a measure of aftermarket frictions that contribute to generating them in practice. Our empirical application uses data from the centralized assignment system for higher education in Chile, which has one of the world’s longest-running college assignment mechanisms based on the deferred-acceptance (DA) algorithm.2 We take advantage of a recent policy change that increased the number of on-platform institutions from 25 to 33, raising the number of available slots by approximately 40%. We first present an analysis of the policy, which shows that when these options are included on the centralized platform, students start college sooner, are less likely to drop out, and are more likely to graduate within 7 years. Importantly, these effects are larger for students from lower-socioeconomic-status (SES) backgrounds, suggesting that the design of platforms can have effects on both efficiency and equity.

...

"We find that when students are allowed to express their preferences for a larger variety of options on the platform, welfare increases substantially, as does the share of students graduating on time.

...

"Intuitively, when a desirable program is not on the platform, it can cause some students who would have placed in that program to instead receive a placement in a different program available on the platform. These students may then decline that placement in favor of the off-platform program, creating vacancies that in turn lead to increased reliance on wait lists, which may be subject to frictions. Moreover, the absence of a particular program may distort the placements of other students, even if the students whose placements are affected would never enroll in that program. These students may also be less satisfied and more likely to decline their placement.

"Taken together, our results show empirically that the existence of off-platform options affects the equity and efficiency of centralized assignment systems. "

############

Earlier (with some links to still earlier papers):

Monday, May 9, 2022

School choice with outside options , by Akbarpour, Kapor, Neilson, van Dijk, and Zimmerman in J.Pub.E.

The Path to a Match for Interventional Cardiology Fellowships

The Society for Cardiovascular Angiography & Interventions has started a fellowship match, and here's an article describing the familiar marketplace failure that led to that decision, involving unraveling of application, interview and appointment dates, with the resulting congestion and exploding offers, and the process of reaching sufficient consensus to move to a centralized match ( to be run by the NRMP).

The Path to a Match for Interventional Cardiology Fellowship: A Major SCAI Initiative  by Douglas E. Drachman MD, FSCAI (Chair) a, Tayo Addo MD b, Robert J. Applegate MD, MSCAI c, Robert C. Bartel MSc, CAE d, Anna E. Bortnick MD, PhD, MSc, FSCAI e, Francesca M. Dea d, Tarek Helmy MD, MSCAI f, Timothy D. Henry MD, MSCAI g, Adnan Khalif MD, FSCAI h, Ajay J. Kirtane MD, SM, FSCAI i, Michael Levy MD, MPH, FSCAI j, Michael J. Lim MD, MSCAI k, Ehtisham Mahmud MD, MSCAI l, Nino Mihatov MD, FSCAI m, Sahil A. Parikh MD, FSCAI i, Laura Porter CMP d, Abhiram Prasad MD n, Sunil V. Rao MD, FSCAI o, Louai Razzouk MD, MPH, FSCAI o, Samit Shah MD, PhD, FSCAI p, Adhir Shroff MD, MPH, FSCAI q, Jacqueline E. Tamis-Holland MD, FSCAI r, Poonam Velagapudi MD, FSCAI s, Fredrick G. Welt MD, FSCAI t, J. Dawn Abbott MD, FSCAI (Co-Chair), Journal of the Society for Cardiovascular Angiography & Interventions, in press.

"Abstract: The field of interventional cardiology (IC) has evolved dramatically over the past 40 years. Training and certification in IC have kept pace, with the development of accredited IC fellowship training programs, training statements, and subspecialty board certification. The application process, however, remained fragmented with lack of a universal process or time frame. In recent years, growing competition among training programs for the strongest candidates resulted in time-limited offers and high-pressure situations that disadvantaged candidates. A grassroots effort was recently undertaken by a Society for Cardiovascular Angiography & Interventions task force, to create equity in the system by establishing a national Match for IC fellowship. This manuscript explores the rationale, process, and implications of this endeavor."

"over the past several years program directors and candidates found that the process has devolved, with wide variation in application timelines and on-the-spot offers, which disadvantage candidates and programs looking to interview a range of applicants.

"The pressures and unfair features of the existing system were further fueled by the transition to virtual interviews related to the COVID-19 pandemic. With logistics of travel no longer a consideration, programs could commence interviews nearly immediately after the applications became available. This led to more candidates being interviewed in rapid succession, and a system evolved in which programs quickly assessed candidates, offered positions, and applied pressure for candidates to accept offers or be passed over for other candidates.

"In response to the shortcomings of the current system, members of Society for Cardiovascular Angiography & Interventions (SCAI) were inspired to lead a grassroots educational campaign to organize IC program directors and the broader interventional community to commit to a regulated “Match” process under the established National Resident Match Program (NRMP). This manuscript provides an account of how this process unfolded and how a Match for IC fellowship was ultimately created.

...

"From the applicant’s perspective, the lack of a structured timeline for the application process required candidates to make career decisions early in the first year of cardiovascular disease training and to compose their application materials 2 years in advance of starting IC training. With ERAS open to application submission in the fall of the second year for the December release to programs, fellows had limited time on clinical rotations to determine their interest and aptitude for IC. Additionally, letters of recommendation, written at this early stage, risked not being fully reflective of each candidate’s capacity to improve and develop the technical skills and clinical knowledge important for success in the field. There were other disadvantages to candidates in the existing system. Fellows at programs with an IC fellowship had an advantage of securing an internal spot but were often pressured to limit their exploration of the opportunities at other programs, potentially disadvantaging them in the long term.

"Another problem with the existing system was that the pressure to recruit candidates on a tight timeline limited the opportunity to interview applicants from a wide variety and diversity of programs, potentially reducing the ability to recruit underrepresented candidates from varied programs. Despite an overall increase in the diversity of physicians entering the workforce,11 there has been little change in the applicant pool for IC over the years, with fewer than 5% of applicants self-reporting as Black race or Hispanic ethnicity and only 10% identifying as women.12

"Competition among the programs, each vying for the seemingly strongest candidates, degenerated into a system that favored quick decision-making on the part of programs to offer positions as early as possible. The influence of the COVID-19 pandemic in 2020 and 2021 negatively impacted an already high-pressure application process, compounding its many weaknesses.13 Fellowship interviews were hosted virtually rather than in person, which enabled candidates to interview at a greater number of programs without the need to travel. In addition, the virtual format accelerated the tempo of an application process that was already felt to be too fast, resulting in an increase in so-called “exploding offers”—offers that required the accepted candidate to respond within a very short timeframe or risk losing the offer. This practice placed significant pressure on candidates to make quick decisions, often forcing them to determine whether to accept the offer from 1 institution before having the opportunity to participate in interviews with—let alone see and evaluate—other programs or fully understand the ramifications of accepting an offer on their personal lives. At the same time, the accelerated timetable left many programs scrambling to identify applicants, as the number of available candidates diminished rapidly due to applicants accepting time-sensitive, exploding offers.

...

"As with other national efforts of this magnitude, the path to develop consensus in favor of a Match was not without challenges. There were several program directors around the country who strongly opposed the institution of a Match. These were well-regarded academicians and clinician educators who expressed very sincere concerns about the impact on fellows in their programs. The members of the SCAI Match Task Force addressed as many concerns as possible, providing the information necessary for each program director to make the best decision for their institution. A minority of program directors remained opposed to the initiative or did not engage with Task Force members despite multiple attempts to be contacted.

"The Match campaign proved highly effective, and by November 2022, the 75% threshold of programs and positions to implement the Match was met

...

"As the sponsor of the Match, SCAI considered the pros and cons of the “All In Policy,” where registered programs must attempt to fill all ACGME positions at the program through the Match.15,16 SCAI opted out of the “All In Policy” to allow programs to have flexibility for unique situations that require commitment to a candidate outside of the Match. 

...

"As a result of the successful implementation of the Match in IC, the first Match cycle for incoming IC fellows will open in the summer of 2024. Individuals eligible to apply include cardiovascular disease fellows in their third or final year of training and graduates who have completed fellowship and are in clinical practice. This class will start IC training in July 2025"

NRMP Board of Directors (2020-2024), and a 40th anniversary

 This bit of glass marks the end of my term on the National Resident Matching Program (NRMP) Board of Directors.

One of the issues that consumed a lot of attention during my term is discussed in this post:

Friday, April 21, 2023

Transition from medical school to residency: defending the parts that work well (namely the NRMP Resident Match)

And here are all my posts about residents and fellows, going back to the beginning of this blog in 2008. (It's been interesting watching medical specialties begin to develop signaling in ways  reminiscent of signaling in the Economics job market, to deal with congestion of interviews and applications.*)

My first paper dealing explicitly with The Match suddenly seems to have been published 40 years ago:

Roth, A.E. "The Evolution of the Labor Market for Medical Interns and Residents: A Case Study in Game Theory", Journal of Political Economy, Vol. 92, 1984, 991‑1016. http://web.stanford.edu/~alroth/papers/evolut.pdf 

And the main report (with Elliott Peranson) of our redesign of The Match is now a quarter of a century old:

Roth, A.E. and E. Peranson, "The Redesign of the Matching Market for American Physicians: Some Engineering Aspects of Economic Design,” American Economic Review, 89, 4, September, 1999, 748-780. https://www.aeaweb.org/articles?id=10.1257/aer.89.4.748

*See yesterday's post for some discussion of market design interventions in job markets.

Social Media and Job Market Success: A Field Experiment on Twitter, by Qiu, Chen, Cohn, and Roth

 Here's a new working paper on SSRN:

Social Media and Job Market Success: A Field Experiment on Twitter, by Jingyi Qiu, Yan Chen, Alain Cohn, and Alvin E. Roth, May 20, 2024

"Abstract: We conducted a field experiment on Twitter to examine the impact of social media promotion on job market outcomes in economics. Half of the 519 job market papers tweeted from our research account were randomly assigned to be quote-tweeted by prominent economists. Papers assigned to be quote-tweeted received 442% more views and 303% more likes. Moreover, candidates in the treatment group received one additional flyout, with women receiving 0.9 more job offers. These findings suggest that social media promotion can improve the visibility and success of job market candidates, especially for underrepresented groups in economics such as women."

I gather that our paper has gone somewhat viral on twitter, with discussion about whether field experiments on job markets are ethical.  That's not a bad discussion to have, and of course we discussed that in the course of planning this experiment. (A similar discussion can and should be be had about any intervention in a market, not just an experiment.*)

Here is what we had to say about that in the paper.

"Despite the positive outcomes, one might question the ethics of our intervention, which randomly promotes a subset of JMPs on social media. However, we observe that senior economists naturally promote their own students and coauthors on Twitter. In comparison, we tweeted every JMP in our sample from our dedicated research account. Furthermore, while 80% of the influencers in our sample come from top 30 institutions, they quote-tweeted JMPs from a broader spectrum of academic institutions, thus allocating attention more equitably. Given that 92% of the JMCs in both the treatment and control groups accepted a job, it is unlikely that our treatment displaced those in the control group. The current focus on diversity, equity, and inclusion suggests that highlighting suitable candidates could potentially expand the number of job openings, making the job market for economists not entirely zero-sum. Lastly, the differential benefit of our treatment for women contributes to fostering a more inclusive economics profession. In summary, we argue that the knowledge gained from our experiment outweighs the potential cost."

*I've been involved in several operational (i.e. not experimental) interventions in job markets, including  the job market for new Econ Ph.D.s (e.g. signaling and the scramble):  see 

 Coles, Peter, John H. Cawley, Phillip B. Levine, Muriel Niederle, Alvin E. Roth, and John J. Siegfried, “The Job Market for New Economists: A Market Design Perspective,” Journal of Economic Perspectives, 24,4, Fall 2010, 187-206.  https://www.aeaweb.org/articles?id=10.1257/jep.24.4.187

Interviews by lottery at Queens University School of Medicine

 Overwhelmed by large numbers of applications, and concerned about fairness, Queens University will set admissions thresholds (on grades and exam scores) and then determine by lottery which applicants move to the interview stage.

New admissions process improves equitable access to the Queen’s MD Program

"Queen’s Health Sciences (QHS) is adapting its MD Program admissions process to create a more inclusive entry point for all applicants, minimize systemic barriers to becoming a doctor, and increase student diversity.

"The renewed medical student admissions process will launch this fall, for 2025 admissions, and includes a pathway for lower socioeconomic status (SES) applicants and adjustments to the current Indigenous pathway. A new comprehensive approach to support the recruitment of Black students will launch in a second phase of this admissions change.  

...

"The MD Program will set admissions thresholds for grade point average (GPA), Medical College Admission Test (MCAT), and Casper (a situational judgment test) at levels that align with the potential for predicting success in medical school, without concern for the number of applicants who meet these thresholds. While the MD Program already posts GPA thresholds, as part of this change, MCAT thresholds will be posted as well. All applicants who meet these thresholds will be entered into a lottery to determine who will be invited to interviews. "

HT: Jonah Peranson (of National Matching Services)

Kidney biopsies can be predicted pretty well, which could speed organ allocation

 Here's a big international collaboration suggesting that AI assisted predictions about kidney biopsies are effective, and could speed acceptance of deceased donor organs for transplant.

 "A Machine Learning-Driven Virtual Biopsy System For Kidney Transplant Patients." Nature Communications 15, no. 1 (2024): 554.  by Daniel Yoo, Gillian Divard, Marc Raynaud, Aaron Cohen, Tom D. Mone, John Thomas Rosenthal, Andrew J. Bentall, Mark D. Stegall, Maarten Naesens, Huanxi Zhang, Changxi Wang, Juliette Gueguen, Nassim Kamar, Antoine Bouquegneau, Ibrahim Batal, Shana M. Coley, John S. Gill, Federico Oppenheimer, Erika De Sousa-Amorim, Dirk R. J. Kuypers, Antoine Durrbach, Daniel Seron, Marion Rabant, Jean-Paul Duong Van Huyen, Patricia Campbell, Soroush Shojai, Michael Mengel, Oriol Bestard, Nikolina Basic-Jukic, Ivana Jurić, Peter Boor, Lynn D. Cornell, Mariam P. Alexander, P. Toby Coates, Christophe Legendre, Peter P. Reese, Carmen Lefaucheur, Olivier Aubert & Alexandre Loupy

From the discussion:

"In this international, multicohort study of kidney transplant biopsies from 17 worldwide centers including the largest Organ Procurement Organization (OPO) in the USA and labeled by expert kidney pathologists, we derived and validated a virtual biopsy system that uses non-invasive and routinely collected donor parameters to predict kidney histological lesions.

...

"Over the past decade, the use of kidneys from older donors with comorbidities has expanded the pool of kidneys, raising the question of whether pathological examination of donated kidneys could help better characterize organ quality or drive inefficiencies in organ allocation22. Additionally, this biopsy procedure needs to be performed and interpreted by trained experts, which is difficult to implement 24/7 . Furthermore, in the USA, the United Network for Organ Sharing policy for organ allocation, recommends the use of KDPI, day-zero biopsy results, and donor characteristics to assess organ quality before transplantation. Despite the importance, the lost time due to this procedure could be precious when the biopsy result is used for allocation purposes as every additional hour of cold ischemia time is highly associated with worse graft outcomes. Therefore, many centers are discouraged from performing day-zero biopsy because it remains an invasive and time-consuming procedure that could increase cold ischemia time."

 

Increasing kidney transplants by reducing discards of risky kidneys

 Kidneys from deceased donors are too often discarded. Dr. Joshua Mezrich, a transplant surgeon at U. Wisconsin, writes in Stat about how to reduce the rate at which high risk kidneys are discarded (after being on ice for a long time while being rejected by many patients). He proposes that kidneys that can be identified as high risk even before being recovered from the deceased donor  be offered promptly to patients/transplant centers that have indicated a willingness to take them. It would require transplant centers to keep current blood tests available for patients who are candidates for high risk kidneys (who may be candidates in part because they are far from the front of the waiting list...)

Too many donor organs go to waste. Here’s how to get them into the patients who need them  By Joshua Mezrich, Stat, March 2, 2024 

"So here is the fix. High-risk kidneys should immediately be offered to transplant centers that opt into a high-risk program as an open offer to their wait list rather than to a specific patient, on a rotating schedule with weight put on proximity to the donor hospital. Ideally the offer should be made prior to procurement of the organ, with final acceptance once it is removed and anatomy and biopsy results can be reviewed by the accepting surgeon.

"If the biopsies show significant disease and the function of the kidney would be inadequate for a recipient, the receiving center can request both kidneys for a single patient, termed a dual transplant (which has been shown to have good outcomes). If a center accepts a kidney, it can then choose the patient who will benefit the most from the transplant and has a long predicted wait time for a low-risk transplant, with informed consent. That would entail a discussion with the patient about expectations regarding the quality of the kidney, how long and how well it might work, and how much longer they might need to wait for a lower-risk kidney. The ability to match the kidney to a recipient is important, as high-risk kidneys need to go into patients who can tolerate the slow initial function. Centers that opt into the high-risk program will need to maintain an updated list of informed patients who are predicted to benefit from these kidneys, who can be called in as soon an offer becomes available. For them, taking a chance beats remaining on dialysis.

Experiments for organ allocation (an idea whose time may be coming)

 Experiments to improve how deceased donor organs are allocated to waiting patients seem like a good idea...

OPTN Task Force sets goal of achieving 60K transplants by 2026, Jan 26, 2024 

“we need to move quicker, be more responsive, and deliver results for the patients we serve,” said Dianne LaPointe Rudow, DNP, president of the OPTN Board of Directors. “The reality is that while the number of transplants continues to grow, so does the non-use of available organs and allocations of organs out of the intended sequence of offers.

...

"The need is clear. In the case of kidneys, the most transplanted organ, the number of kidneys recovered from deceased donors increased by 56 percent between 2018 to 2023. Yet the number of kidney transplants only increased by 44 percent, meaning that approximately one quarter of kidneys recovered were not transplanted.

...

"Under a proposed variance for expedited placement, currently out for public comment, the task force intends to develop a series of rapid, small-scale tests of innovative organ placement approaches and assess their outcomes to evaluate whether they could be incorporated into future OPTN policies. The task force also has committed to prioritizing studies that evaluate potential frameworks for allocating hard-to-place organs to increase the number of transplants and lower non-use rates."

Pilot projects to speed deceased donor organ allocation

I won't use the word "experiments" for fear of the evil eye, but the OPTN is planning to authorize 'pilot programs' to try to speed the allocation of deceased donor organs. The idea is that after an organ has been rejected numerous times, and has started to be in danger of discard, it can be offered to a transplant center and patient that are likely to accept and transplant it, rather than proceeding to offer it to centers and patients in priority order. This is important, because too much waiting time on ice is one of the chief reasons that organs are rejected and eventually discarded.

Expedited Placement Variance his proposal recommends a new variance related to expedited organ placement and proposes modifying the OPTN’s variance process in order to allow for more rapid studies of potential improvement.

"Proposed changes

"Create a variance to govern pilot projects related to expedited organ placement.

Gives the OPTN Executive Committee authority to develop protocols for expedited organ placement.

This approach will allow the OPTN the ability to rapidly iterate on different protocols.

Update portions of the OPTN’s governance structure regarding variances.

This will allow for a more rapid and iterative approach when creating new variances."

Read the full proposal (PDF)

"The task force intends to conduct multiple iterative pilots or PDSAs with the community to identify effective practices to improve the efficiency of the organ allocation process. (Not all pilots or PDSAs will require a policy variance.) This proposal 1) creates a variance to govern the expedited placement pilots and 2) adjusts the OPTN’s governance of all variances. Additional variances or process improvement projects will focus on other topics to improve the efficiency of the organ allocation process.

"The Committee is issuing this proposal for a thirty-day public comment period, which is shorter than the usual public comment period. This is to allow the variance to take effect sooner than the end of regular public comment but allow the community time to comment on the proposed variance. It also is in line with the public comment periods for emergency and expedited policy changes."

Applying for medical residencies: a consensus statement from Internal Medicine

 The Alliance for Academic Internal Medicine has released a "consensus statement" with many proposals about application and interview caps, and signaling.

Catalanotti, Jillian S., Reeni Abraham, John H. Choe, Kelli A. Corning, Laurel Fick, Kathleen M. Finn, Stacy Higgins et al. "Rethinking the Internal Medicine Residency Application Process to Prioritize the Public Good: A Consensus Statement of the Alliance for Academic Internal Medicine." The American Journal of Medicine (2023).

It also includes a call for data and analysis:

"AAIM proposes increasing internal medicine program preference signals to 15, using tiered signaling with three “gold” and 12 “silver” signals, and setting an interview cap of 15 in the 2024-2025 recruitment season, with participation by all internal medicine programs. The Alliance recommends that all internal medicine programs participate in ACI. AAIM recommends that programs transparently share information about their use of preference signals and other application screening methods and calls for real-time data analysis to explore impact, inform future iterations and identify potential harms.

"The Alliance calls upon ERAS and NRMP as well as Thalamus® and other interview scheduling platforms to transparently share data, to embrace change, and to perform analyses needed to inform this process. For example, recent modeling with eight years of retrospective NRMP data in OBGYN demonstrated that an early match round may increase the number of “mutually dissatisfied applicant-program pairs” and that a multiple-round match process could introduce potential rewards for gamesmanship, a prime factor addressed by the current process.35 AAIM applauds this analysis and hopes that the new collaboration between ERAS and Thalamus® may provide useful interview data to inform this proposal and further interventions."

And here is reference 35 in that last paragraph, about which I've blogged before.

I Ashlagi, E Love, JI Reminick, AE. Roth

Early vs Single Match in the Transition to Residency: Analysis Using NRMP Data From 2014 to 2021

J Grad Med Educ, 15 (2) (Apr 2023), pp. 219-227, 10.4300/JGME-D-22-00177.1

View in ScopusGoogle Scholar

Congestion in peer review, and an experiment in Nature (the journal)

 Too many papers, and not enough qualified reviewers willing or able to spend the time reviewing them.  And when a paper is reviewed, revised and published, there may have been valuable scientific discussions that went on in private

Nature has a column about the first problem, and is conducting an experiment with published papers that might help alleviate the second.

Here's the column about congestion:

Stop the peer-review treadmill. I want to get off. Faced with a deluge of papers, journal editors are struggling to find willing peer reviewers. by Amber Dance

"In November 2022, health economist Chris Sampson found himself in desperate need of a hero. As associate editor for Frontiers in Health Services, he’d been trying to get a paper reviewed since April. He’d sent out about 150 invitations to potential reviewers and received four reviews, but only one of sufficient quality to be useful.

...

"Sampson’s plight is a problem faced by editors the world over in the face of uncontrolled growth in the number of journals and papers. The tally of articles indexed by the citation database Web of Science tripled from about one million in 1990 to nearly three million in 2016, according to the website Publons

...

"Balazs Aczel is a psychologist at Eötvös Loránd University in Budapest who studies the processes of science. Using a data set covering more than 87,000 scholarly journals, Aczel and his colleagues estimated that researchers globally, in aggregate, spent the equivalent of more than 15,000 years on peer review in 2020 alone2. And many scientists are declining to review more frequently.

...

"Many scientists are increasingly frustrated with journals — Nature among them — that benefit from the unpaid work of reviewing while charging high fees to publish in them or read their content. 

...

"Yates and others suggest that cash payments would solve the problem, but others say such a system would be unethical and unsustainable."

###########

And here's an editorial about the experiment involving published papers and their referee reports (but not, apparently, involving unpublished papers):

Nature is trialling transparent peer review — the early results are encouraging. Last year, nearly half of Nature authors agreed to publish anonymous referee reports. We hope that more will consider doing so this year.  EDITORIAL, Nature, 01 March 2022

"Research papers are the product of lengthy discussions between authors and reviewers — guided by editors. These peer-review conversations can last for months at a time and are essential to progress in research. There is widespread agreement that the robustness and clarity of papers are enhanced in this process.

"Peer-review exchanges are mostly kept confidential, meaning that the wider research community and the world have few opportunities to learn what is said in them. Such opacity can fuel perceptions of secrecy in publishing — and leaves reviewers and their key role in science publication underappreciated. It also robs early-career researchers of the opportunity to engage with examples of the inner workings of a process that is key to their career development.

"In an attempt to change things, Nature Communications has since 2016 been encouraging authors to publish peer-review exchanges. In February 2020, and to the widespread approval of Twitter’s science community, Nature announced that it would offer a similar opportunity. Authors of new manuscript submissions can now have anonymous referee reports — and their own responses to these reports — published at the same time as their manuscript. Those who agree to act as reviewers know that both anonymous reports and anonymized exchanges with authors might be published. Referees can also choose to be named, should they desire.

"A full year’s data are now in, and the results are encouraging. During 2021, nearly half (46%) of authors chose to publish their discussions with reviewers, although there is variation between disciplines 

...

"The benefits to research are huge. Opening up peer review promotes more transparency, and is valuable to researchers who study peer-review systems. It is also valuable to early-career researchers more broadly. Each set of reports is a real-life example, a guide to how to provide authors with constructive feedback in a collegial manner.

Publishing peer-review exchanges, in addition, recognizes the effort that goes into the endeavour. Peer review is integral to being a researcher. Making reviewers’ work public illustrates the lengths that researchers will go in the service of scholarship. According to one study, reviewers in total do tens of millions of hours of peer review each year (B. Aczel et al. Res. Integr. Peer Rev. 6, 14; 2021). Yet this contribution is rarely recognized in research evaluation systems. As we have reported, there is growing interest in reforming these systems to better represent how science is done. If more researchers agree to open up their peer-review exchanges, we can all play a part in making that happen.

Nature 603, 8 (2022)