This bit of glass marks the end of my term on the National Resident Matching Program (NRMP) Board of Directors.
One of the issues that consumed a lot of attention during my term is discussed in this post:
I'll post market design related news and items about repugnant markets. See also my Stanford profile. I have a general-interest book on market design: Who Gets What--and Why The subtitle is "The new economics of matchmaking and market design."
This bit of glass marks the end of my term on the National Resident Matching Program (NRMP) Board of Directors.
One of the issues that consumed a lot of attention during my term is discussed in this post:
The Alliance for Academic Internal Medicine has released a "consensus statement" with many proposals about application and interview caps, and signaling.
Catalanotti, Jillian S., Reeni Abraham, John H. Choe, Kelli A. Corning, Laurel Fick, Kathleen M. Finn, Stacy Higgins et al. "Rethinking the Internal Medicine Residency Application Process to Prioritize the Public Good: A Consensus Statement of the Alliance for Academic Internal Medicine." The American Journal of Medicine (2023).
It also includes a call for data and analysis:
"AAIM proposes increasing internal medicine program preference signals to 15, using tiered signaling with three “gold” and 12 “silver” signals, and setting an interview cap of 15 in the 2024-2025 recruitment season, with participation by all internal medicine programs. The Alliance recommends that all internal medicine programs participate in ACI. AAIM recommends that programs transparently share information about their use of preference signals and other application screening methods and calls for real-time data analysis to explore impact, inform future iterations and identify potential harms.
"The Alliance calls upon ERAS and NRMP as well as Thalamus® and other interview scheduling platforms to transparently share data, to embrace change, and to perform analyses needed to inform this process. For example, recent modeling with eight years of retrospective NRMP data in OBGYN demonstrated that an early match round may increase the number of “mutually dissatisfied applicant-program pairs” and that a multiple-round match process could introduce potential rewards for gamesmanship, a prime factor addressed by the current process.35 AAIM applauds this analysis and hopes that the new collaboration between ERAS and Thalamus® may provide useful interview data to inform this proposal and further interventions."
And here is reference 35 in that last paragraph, about which I've blogged before.
The market for new doctors has been suffering from congestion in applications and interviews, in the runup to the resident Match (see recent post with a diagram). The American Association of Medical Colleges runs the main application server, ERAS. A private company called Thalamus runs a growing interview scheduling service. Now they are looking to collaborate.
Here's yesterday's press release from Thalamus:
Collaboration will increase transparency and make the residency process easier for applicants and programs
Washington, D.C., April 27, 2023—Today the AAMC (Association of American Medical Colleges) and Thalamus announced a strategic collaboration to accelerate innovation and ease the transition to residency for medical students, medical schools, and residency programs. The collaboration will combine the AAMC’s long-established leadership in innovation along the continuum from medical school to residency training and continuing medical education with Thalamus’ market-leading product and software development expertise.
“The transition from undergraduate medical education to graduate medical education is a critical period in any learner’s journey to becoming a physician,” said David J. Skorton, MD, AAMC president and CEO. “We know the community is seeking enhanced tools and integrated services that better support application and recruitment processes. We listened, we have made improvements, and, with Thalamus, we are excited to make this vision a reality.”
The organizations will collaborate to leverage their data, technology, and expertise to transform the medical residency and fellowship recruitment processes for applicants and programs. Their efforts will focus on increasing transparency, supporting equity through holistic review, and improving the learner experience by consolidating the fragmented interview management process.
“We are thrilled to be collaborating with the AAMC to provide a comprehensive solution that will streamline graduate medical education recruitment processes,” said Jason Reminick, MD, MBA, MS, CEO and founder of Thalamus. “But even more, we are looking forward to building new and innovative tools that improve the experience, are cost-effective, and leverage data for the benefit of the medical education community and the advancement of our collective missions.” Dr. Reminick applied to residency in 2012 during an eventful recruitment season disrupted by Hurricane Sandy. “I’m particularly excited to provide applicants with a comprehensive platform to manage their interview season.”
The collaboration between the AAMC and Thalamus will enable data-sharing and innovative research that will benefit the undergraduate to graduate medical education community and advance both organizations’ missions. The initiative also demonstrates the commitment of both organizations to addressing the concepts and themes outlined in the 2021 report from the Coalition for Physician Accountability’s Undergraduate Medical Education-Graduate Medical Education Review Committee.
In recent years, the AAMC has completed significant in-depth research and upgraded technology to enhance the Electronic Residency Application Service® (ERAS®) suite of application and selection tools, such as updating the MyERAS® application content, building analytics tools for institutions, and partnering on collaborative research initiatives. Thalamus has completed unique research related to the physician workforce, including how geography influences The Match® and specialty-specific interview practices. The Thalamus technology will continue the upgrade of the ERAS suite of application and selection tools. The AAMC and Thalamus remain committed to future innovations that will enable the ERAS program to continue to evolve faster and better.
Beginning in June 2023, all ERAS residency and fellowship programs will receive complimentary access to Thalamus’ leading interview management platform, Thalamus Core and Itinerary Wizard, as well as Cerebellum, a novel data and analytics dashboard to assess recruitment outcomes, specifically from a diversity, equity, inclusion, and geographic perspective. Programs may also elect to purchase Thalamus’ video interview platform and Cortex, its technology-assisted holistic application review and screening platform.
According to AAMC data, the U.S. is expected to experience a shortage of up to 124,000 physicians by 2034. Given the burnout and other challenges to the health care system caused by the COVID-19 pandemic, the AAMC and Thalamus look to use their collective expertise to promote a diverse and representative workforce that will enhance health care and patient outcomes.
The data and research the AAMC and Thalamus have amassed to identify resident, fellow, and physician recruitment trends can potentially have a major impact on diversity in medicine and begin to address several well-established and longstanding systemic challenges. These efforts will support not only the application and selection processes in graduate medical education but also aim to improve the experiences of the U.S. physician workforce over the long term.
Related Resources
###
Note for editors: Leaders from the AAMC and Thalamus are available to speak with media about this new collaboration and what it means for residency programs and applicants.
The AAMC (Association of American Medical Colleges) is a nonprofit association dedicated to improving the health of people everywhere through medical education, health care, medical research, and community collaborations. Its members are all 157 U.S. medical schools accredited by the Liaison Committee on Medical Education; 13 accredited Canadian medical schools; approximately 400 teaching hospitals and health systems, including Department of Veterans Affairs medical centers; and more than 70 academic societies. Through these institutions and organizations, the AAMC leads and serves America’s medical schools and teaching hospitals and the millions of individuals across academic medicine, including more than 193,000 full-time faculty members, 96,000 medical students, 153,000 resident physicians, and 60,000 graduate students and postdoctoral researchers in the biomedical sciences. Following a 2022 merger, the Alliance of Academic Health Centers and the Alliance of Academic Health Centers International broadened the AAMC’s U.S. membership and expanded its reach to international academic health centers. Learn more at aamc.org.
Thalamus is the premier, cloud-based interview management platform designed specifically for application to Graduate Medical Education (GME) training programs. The software streamlines communication by eliminating unnecessary phone calls/emails allowing applicants to book interviews in real-time, while acting as a comprehensive applicant tracking system for residency and fellowship programs. Thalamus provides comprehensive online interview scheduling and travel coordination via a real-time scheduling system, video interview platform, AI application screening/review tool (Cortex) providing technology-assisted holistic review, and first-in-class DEI-focused analytics dashboard (Cerebellum). Featured nationally at over 300+ institutions and used by >90% of applicants, Thalamus is the most comprehensive solution in GME interview management. For more information on Thalamus, please visit https://thalamusgme.com or connect with us on LinkedIn, Facebook, Instagram, Twitter, or YouTube.
Here's a paper that investigates two alternatives to limiting congestion in college admissions: one is to limit applications, and the other is to add a small cost for each additional application. (This is a current topic of discussion in a number of other applications, including matching of new doctors to residencies.)
Application Costs and Congestion in Matching Markets by YingHua He and Thierry Magnac, The Economic Journal, https://doi.org/10.1093/ej/ueac038 (online early)
Abstract: "A matching market often requires recruiting agents, or ‘programmes’, to costly screen ‘applicants’, and congestion increases with the number of applicants to be screened. We investigate the role of application costs: higher costs reduce congestion by discouraging applicants from applying to certain programmes; however, they may harm match quality. In a multiple-elicitation experiment conducted in a real-life matching market, we implement variants of the Gale-Shapley deferred-acceptance mechanism with different application costs. Our experimental and structural estimates show that a (low) application cost effectively reduces congestion without harming match quality."
"Our empirical strategy is novel. It begins with a multiple-elicitation field experiment that enables us to directly evaluate the effects of application costs. The experiment involves the real-life matching of 129 applicants to the seven master’s programmes at the Toulouse School of Economics (TSE), and was conducted in May 2013 for admission in the 2013–4 academic year. The experimental market designs are three variants of the Gale-Shapley deferred-acceptance (DA) mechanism encountered in practice: the traditional DA mechanism, under which applicants can apply to all programmes without any cost; the DA mechanism with truncation (DA-T), under which applicants can apply to no more than four programmes (hence, DA-T-4); and the DA mechanism with cost (DA-C), under which applicants must write a motivation letter for each additional application beyond the first three applications. Under each mechanism, every applicant is required to submit a rank-ordered list of programmes (ROL). As applicants are informed that one of the mechanisms will be implemented, they have incentives to behave optimally under each mechanism.
"To evaluate the performance of a matching procedure, we focus on two dimensions of a matching outcome: the congestion and match quality. The former is measured by screening costs and approximated by the number of applicants to screen; the latter is measured by the welfare of both sides, the number of unmatched applicants, as well as the number of blocking pairs. A pair comprising applicant and programme blocks a matching if both would be better off by being matched together after leaving their current matches. The stability of a matching, defined as the absence of any blocking pair, is the key to the success of matching markets (Roth, 1991). Importantly, stability implies Pareto efficiency when both sides are endowed with strict preferences (Abdulkadiroğlu and Sönmez, 2013)."
While there are concerns that electronic application services like the common app have led to an explosion of applications, there are also concerns in the opposite direction, that applying to college remains a barrier particularly to students who don't automatically think of college as an option.
Here's a story from the Chronicle of Higher Ed about an initiative to ease the application process.
Rethinking the Act of Applying to College. A tedious process that puts the onus on students may need an overhaul. By Eric Hoover
"On Thursday, the Coalition for College, a membership group of 162 institutions with a shared online application, announced a plan to ease the challenge of applying. As part of a new partnership, the organization will embed its application process into Scoir, an online college-advising platform used by students at more than 2,000 high schools nationwide.
"Instead of creating a Coalition application and typing information into a separate website, students with a Scoir account will soon be able to apply to any Coalition college by transmitting an admission form prepopulated with information — demographic data, grades, test scores, and so on — that would already reside under the same virtual roof.
...
"The more complex the application process, the less equitable it becomes."
"That was a key line from a report published in January by the National Association for College Admission Counseling and the National Association of Student Financial Aid Administrators. The report, which imagined what the college-application process would look like if racial equity were the main objective, included findings and recommendations drawn from interviews with a panel of admissions and financial-aid experts, as well as students."
Medpage Today has the story, which has been active on medical twitter for a while--some residency candidates are finding that their applications are incomplete, and blame the common application service ERAS, which in response reports that incomplete applications are the result of errors by the candidates.
"MedPage Today spoke to several residency applicants who believe they experienced ERAS tech issues when submitting their applications. A majority of these applicants requested anonymity due to fear of retribution.
While many say a potential bug is not surprising, students are frustrated that they cannot pin down when "their documents went missing -- and whether or not it jeopardized their chances of matching with some programs.
"In each case, the glitch is almost identical: an applicant's letters of recommendation (or other documents) were uploaded to ERAS, and the applicant was certain they assigned the right materials to each program before they hit submit. But weeks later, applicants checked the platform to see that these documents had never been assigned nor delivered to the right programs -- rendering their applications incomplete."
************
Related:
Med Students Filed Nearly 70 Residency Applications Each— "I don't think this is sustainable at all" by Amanda D'Ambrosio, MedPage Today October 27, 2021
"Across all medical specialties, this cycle MD applicants submitted an average of 68 applications, DO applicants submitted 92, and international medical graduates submitted 139, according to data provided to MedPage Today by the Association of American Medical Colleges (AAMC).
"Medical graduates applying to orthopedic surgery in the 2022 cycle submitted an average of 88 applications -- the highest across all medical specialities. Candidates to urology, otolaryngology, and dermatology -- some of the most competitive specialties -- all applied to around 80 programs each.
"More than 46,000 medical graduates have applied for the 2022 residency match so far: approximately 23,000 MDs, 8,000 DOs, and 15,000 international graduates. The preliminary data from ERAS -- the centralized system that medical graduates use to apply to residency -- includes all applications submitted through the beginning of October."
Here's a paper modeling the issue that some labor markets may face congestion related to large numbers of applications followed by costly interviews.
Nick Arnosti, Ramesh Johari, Yash Kanoria (2021) Managing Congestion in Matching Markets. Manufacturing & Service Operations Management 23(3):620-636. https://doi.org/10.1287/msom.2020.0927
Abstract. "Problem definition: Participants in matching markets face search and screening costs when seeking a match. We study how platform design can reduce the effort required to find a suitable partner. Practical/academic relevance: The success of matching platforms requires designs that minimize search effort and facilitate efficient market clearing.
"Methodology: We study a game-theoretic model in which “applicants” and “employers” pay costs to search and screen. An important feature of our model is that both sides may waste effort: Some applications are never screened, and employers screen applicants who may have already matched. We prove existence and uniqueness of equilibrium and characterize welfare for participants on both sides of the market. Results: We identify that the market operates in one of two regimes: It is either screening-limited or application-limited. In screening-limited markets, employer welfare is low, and some employers choose not to participate. This occurs when application costs are low and there are enough employers that most applicants match, implying that many screened applicants are unavailable. In application-limited markets, applicants face a “tragedy of the commons” and send many applications that are never read. The resulting inefficiency is worst when there is a shortage of employers. We show that simple interventions—such as limiting the number of applications that an individual can send, making it more costly to apply, or setting an appropriate market-wide wage—can significantly improve the welfare of agents on one or both sides of the market.
"Managerial implications: Our results suggest that platforms cannot focus exclusively on attracting participants and making it easy to contact potential match partners. A good user experience requires that participants not waste effort considering possibilities that are unlikely to be available. The operational interventions we study alleviate congestion by ensuring that potential match partners are likely to be available.
And from the Conclusion:
"We also compare the effects of an application limit to those of other available levers: either raising application costs or lowering the wage paid to applicants. Although these interventions can lead to thesame aggregate welfare as an application limit, they differ in how they distribute this welfare. Charging fees and lowering wages both increase aggregate welfare at the expense of applicants. Although these interventions may be appropriate for a platform looking to monetize its services or attract more employers, an application limit can yield Pareto improvements in welfare and may be more suitable if the platform is primarily concerned with applicant welfare. These considerations might explain why the tutoring platform TutorZ charges tutors for each potential client that they contact, whereas the dating platforms Coffee Meets Bagel and Tinder limit the number of likes/right swipes permitted in a certain period."
A pair of papers study the Common App, how it is used disproportionally by selective universities and liberal arts colleges, to which applications have increased over time. The papers focus on how this has increased student choice.
There's a parallel set of arguments made elsewhere, particularly in connection with application to medical residencies, that too many applications increase congestion in the admissions process.
The Common Application and Student Choice, By Brian Knight and Nathan Schiff, AEA Papers and Proceedings 2021, 111: 460–464, https://doi.org/10.1257/pandp.20211042
And here's a longer companion paper:
Reducing Frictions in College Admissions: Evidence from the Common Application by Brian Knight and Nathan Schiff, April 17, 2020
Abstract: College admissions in the U.S. is decentralized, creating frictions that limit student choice. We study the Common Application (CA) platform, under which students submit a single application to member schools, potentially reducing frictions and increasing student choice. The CA increases the number of applications received by schools, reflecting a reduction in frictions, and reduces the yield on accepted students, reflecting increased choice. The CA increases out-of-state enrollment, especially from other CA states, consistent with network effects. CA entry changes the composition of students, with evidence of more racial diversity, more high-income students, and imprecise evidence of increases in SAT scores.
Covid forced lots of colleges to make standardized tests optional in admissions, and that seems to have jolted the growth in college applications to new highs. The Chronicle of Higher Education has the story:
The Endless Sensation of Application Inflation By Eric Hoover
"consider a big-deal development: the suspension of standardized-testing requirements. After most of the nation’s big-name colleges adopted test-optional policies for the 2020-21 cycle, they all but guaranteed a surge in applications from students who otherwise wouldn’t have applied. When that surge came, some admissions deans publicly expressed surprise that their testing requirements apparently had been suppressing applications from underrepresented students all along, just as critics of ACT and SAT requirements have been saying for decades.
...
"there are some drawbacks to having an overwhelming number of choices, Brennan says: “In admissions, you don’t get a 20-percent increase in staff to account for a 20-percent increase in applications.”
Here's an essay from the WSJ, adapted from a forthcoming book with an evocative title, “Who Gets In and Why, by Jeffrey Selingo. The subtitle is A Year Inside College Admissions
The Secrets of Elite College Admissions: In the final ‘shaping’ of an incoming class, academic standards give way to other, more ambiguous factors by By Jeffrey Selingo, Aug. 28, 2020
"The year I was inside Emory University’s admissions office, the school received a record 30,000 applications for fewer than 1,400 spots in its incoming class. In early March, just weeks before official notices were scheduled to go out, the statistical models used by Emory to predict enrollment indicated that too many applicants had been chosen to receive acceptances. In the span of days, teams of admissions officers covering five geographical areas had to shift 1,000 applications from the thin “admit” stack to the much larger “deny” or “wait list” piles.