Cleaning up the scramble for medical residents with SOAP

Changes are coming in the scramble that follows the medical resident match run by the National Resident Matching Program (NRMP). A new, semi-centralized, partially computerized Supplemental Offer and Acceptance Program (SOAP) has been announced, for implementation in March 2010.

If I understand the proposal correctly, residency programs with unmatched positions will be able to submit preference lists of match participants who ended up unmatched, and these preferences will be used to make exploding offers, after which the preference lists and positions will be updated to take account of acceptances (e.g. candidates who accept a position will be removed from other program's preference lists), and new exploding offers will be issued. Unlike in the main match (which these days uses the Roth-Peranson algorithm, but which you can think of as a student-proposing deferred acceptance algorithm), applicants will not submit preference lists, but will accept or reject offers as they come in (i.e. they cannot defer acceptances by holding their best offer until they see if any better offers arrive later).

Part of the proposal is to integrate the scramble with the Electronic Residency Application Service (ERAS), which will make some kinds of automatic processing and regulation possible,while some of the proposed regulations may be more challenging to enforce. (Markets that use exploding offers at fixed times have often been subject to cheating of various sorts: here's a paper on the experience of the law clerk market. There are lots of differences in the market culture of doctors and lawyers that may result in different outcomes.)

Here are some of its proposed rules:

• Unmatched applicant and unfilled program information will be released simultaneously.

• There will be a “time out” period during which unmatched applicants can send applications but programs cannot make offers.

• Applicants and programs will be required to send and receive applications only through ERAS.

• NRMP-participating programs that fill positions during Match Week must do so only through the SOAP.

• New functionality will be added to the R3 System to allow programs to offer unfilled positions on the basis of preference lists submitted by the programs.

• Applicants must accept or reject their offer(s) within a specific timeframe; offers not accepted or rejected will expire.

• The R3 System will establish an electronic “handshake” when an applicant accepts a position.

• Positions will be deleted from the dynamic List of Unfilled Programs once an offer has been accepted.

• A program’s unfilled positions will be offered to applicants in order of preference until all positions are filled or the preference list has been exhausted; programs will be able to add applicants to the bottom of their preference lists throughout Match Week.

• The NRMP Match Participation Agreement will be expanded to include Match Week and SOAP, and sanctions will be imposed for improper behavior.
...
Eligible NRMP applicants:


• Must be able to enter GME on July 1 in the year of the Match

• Will be able to apply only to unfilled Match-participating programs during Match Week

􀀹 Access to the List of Unfilled Programs will be restricted by match status (preliminary or advanced)

􀀹 Must use ERAS and will be able to select only unfilled Match-participating programs

􀀹 Cannot use phone, fax, email, or other methods

􀀹 Cannot have another individual/entity contact programs on applicant’s behalf

􀀹 Will be able to accept positions only through SOAP during Match Week

• Can apply to non-Match-participating programs after Match Week

Ineligible NRMP applicants:

• Cannot participate in SOAP

􀀹 Cannot apply to Match-participating programs using ERAS, phone, fax, email, or other methods

􀀹 Cannot have another individual/entity contact Match-participating programs on applicant’s behalf

• Can apply to non-Match-participating programs during Match Week

􀀹 Can use ERAS to select non-Match-participating programs

􀀹 Can use phone, fax, email, or other methods

• Can apply to Match-participating programs after Match Week

Unfilled Programs:

• Must accept applications only through ERAS during Match Week

􀀹 Cannot use phone, fax, email, or personal contacts

• Must fill positions using SOAP during Match Week

􀀹 Cannot offer positions to ineligible applicants during Match Week

􀀹 Cannot make offers outside SOAP during Match Week

􀀹 Are not required to fill positions during Match Week

• Can add applicants to bottom of preference list

If an applicant rejects an offer or allows an offer to expire, no further offers will be made to that applicant by the same program.

Once an applicant accepts an offer, the applicant will not be able to send additional applications via ERAS.

Once a program has filled all of its positions through SOAP, applicants will be unable to send applications to that program via ERAS.

Offers extended by programs and accepted by applicants during the Match Week Supplemental Offer and Acceptance Program (SOAP) will create a binding commitment. Failure to honor that commitment or failure to adhere to SOAP policies will be a violation of the Match Participation Agreement.

2013 Results and Data Book from NRMP’s Main Residency Match

Here's the press release: 2013 Results and Data Book from NRMP’s Main Residency Match, and here's the data book for the 2013 Match

The data book also reports on the second year of the new SOAP scramble, but not in the same detail as last year's report. Last year I predicted that more of the visible action would move to the first round (reflecting more action before the beginning of the official scramble), but this year's data book doesn't give that information.

Here's the press release, though, which gives a good overview of the data, which seem to reflect a successful match year.

The National Resident Matching Program® (NRMP®) is pleased to announce the publication of its Results and Data 2013 Main Residency Match --- the most comprehensive Match data resource produced annually by the NRMP. The report provides statistics on the most successful Match in NRMP history, in which 28,130 of 29,171 available residency positions were filled.    

Included in the report are statistical tables and data graphs from the Main Residency Match and a state by state breakdown of each participating U.S. residency training program, with the number of positions offered and filled for each. This year, for the first time, the report includes results and charts from the Match Week Supplemental Offer and Acceptance Program® (SOAP®)-- the process used by NRMP for unfilled residency positions.

This year’s Results and Data Book is notable because the 2013 Match was the first to utilize NRMP’s All In Policy, which requires participating programs to register and attempt to fill all their positions through The Match.

NRMP Executive Director Mona M. Signer said, “Readers of the 2013 Results and Data Book will see the overwhelming success of the All In Policy. The 2013 Match produced the highest fill rate in NRMP history, and match rates rose for nearly every applicant group.”

Results and Data 2013 Book Facts

Applicant Numbers (Comparisons to 2012) 

• 40,335 registrants (1,958 more)
• 34,355 active applicants (3,000 more)
• 17,487 U.S. allopathic medical school seniors (960 more)
• 2,677 osteopathic medical school students/graduates (317 more, highest ever)
• 5,095 U.S. citizen international medical school students/graduates (U.S. IMGs) (816 more)
• 7,568 non-U.S. citizen international medical school students/graduates (IMGs) (740 more)

Applicant Match Rates 

• 74.1 percent of all applicants were matched to PGY-1 positions
• 93.7 percent of U.S. allopathic seniors matched to PGY-1 positions; of those, 78.8 percent matched to one of their top three choices
• 53.1 percent of U.S. IMGs were matched to PGY-1 positions, up from 49.1 percent in 2012 and the highest Match rate for this applicant group since 2005
• 47.6 percent PGY-1 Match rate for IMGs, an increase of seven percentage points from 2012

Increased Positions/High Fill Rates
Available residency positions increased to 29,171, 2,399 (9 percent) more than in 2012, when 26,772 positions were offered. The increase is due primarily to implementation of the All In Policy, which resulted in some specialties offering significantly more positions in 2013. Internal Medicine offered 1,000 more positions, Family Medicine offered 297 more, and Pediatrics offered 141 more than the prior year.

The overall position fill rate increased 1.1 percentage points to 96.4 percent, the highest in NRMP history. In 2013, only 1,041 first-year and second-year positions were unfilled; of those, 939 were placed in SOAP, and all but 61 were filled.

Notable Specialties
Often Match results can be an indicator of competitiveness. This year, several specialties and specialty tracks had 100 percent fill rates: 

• Medicine Emergency Medicine
• Pediatrics Primary
• Pediatrics/Psychiatry/Child Psychiatry
• Physical Medicine and Rehabilitation – Post graduate year one (PGY-1)
• Plastic Surgery – Post graduate year two (PGY-2)
• Psychiatry Family Medicine
• Radiation Oncology (PGY-1)
• Thoracic Surgery

The following specialties filled more than 90 percent of positions with U.S. seniors: 

• Plastic Surgery: 95.7 percent
• Otolaryngology: 94.5 percent
• Neurological Surgery: 93.1 percent
• Orthopedic Surgery: 91.8 percent

Couples in the Match
In 2013, a record 1,870 applicants (935 couples) participated in the 2013 Match. They continued to enjoy great success with a match rate of 95.2 percent. Couples have been able to participate together in The Match since 1984. The two partners identify themselves as a couple to the NRMP and submit rank order lists of identical length. The algorithm treats their lists as a unit, matching the couple to the highest linked program choices where both partners match.

SOAP - Supplemental Offer and Acceptance Program
The NRMP launched the Match Week SOAP in 2012 to replace the "Scramble" and to streamline the process for unmatched applicants and the directors of unfilled programs. During SOAP, eligible applicants use ERAS® to apply to programs with unfilled positions and offers are extended to applicants based upon the number of unfilled positions remaining in the program.

U.S. seniors accepted the majority of positions (595 of 878) filled during SOAP. More than 13,808 applicants were SOAP-eligible, 261 more than in 2012. This year, 406 unfilled programs elected to participate in SOAP, offering 939 of the 1,041 positions not filled when the matching algorithm was processed.

A total of 1,327 offers were sent to applicants. By the conclusion of SOAP, 93.5 percent (878 of 939) of the positions had been filled.

How the Match Works
Conducted annually by the NRMP, The Match uses a computerized mathematical algorithm to align the preferences of applicants with the preferences of residency program directors in order to fill the training positions available at U.S. teaching hospitals. Research on the NRMP algorithm was a basis for Dr. Alvin Roth’s receipt of the 2012 Nobel Prize in Economics.

# # #
The National Resident Matching Program® (NRMP®) is a private, non-profit organization established in 1952 at the request of medical students to provide an orderly and fair mechanism for matching the preferences of applicants for U.S. residency positions with the preferences of residency program directors. In addition to the Main Residency Match, the NRMP conducts matches for more than 40 subspecialties. For more information, contact NRMP at 1-866-653-NRMP (6767) or visit http://www.nrmp.org. For interviews, please email cherbert@nrmp.org.

Read more: http://www.digitaljournal.com/pr/1278215#ixzz2V7ttPAqX

Comment on the proposed NRMP scramble following the resident match

The NRMP website asked for comments on the NRMP's new plan for organizing the post-match scramble, and some of my young colleagues and I were moved to send one in:

Comment on the NRMP’s “Supplemental Offer and Acceptance Program” Proposed to Replace the Post-Match Scramble by Peter A. Coles, Clayton R. Featherstone, John William Hatfield, Fuhito Kojima, Scott Duke Kominers, Muriel Niederle, Parag A. Pathak, and Alvin E. Roth.

Executive Summary: "Historical precedent and economic principles suggest that the Supplemental Offer and Acceptance Program (SOAP) proposed for the NRMP Scramble will lead to unsatisfactory outcomes by forcing participants to make unnecessarily difficult decisions and giving them strong incentives to break the rules laid out in the SOAP proposal. We suggest, as an alternative Scramble mechanism, that the NRMP run a “Second Match” for the Scramble participants using rules similar to those of the Main Match."


Here's my previous post: Cleaning up the scramble for medical residents with SOAP

Medical residency scramble

Dr.  Naveed Saleh, writing in New Physician, describes some of the different prognoses suggested for the new resident scramble (SOAP) which will operate next year: Unscrambling the Match.

"Roth predicts that the rules of the SOAP will be subverted by both programs and applicants eager to match. "If it's really, really tempting for people on both sides to break the rules," says Roth, "often the rules get broken."
Roth suggests that instead of the SOAP, the NRMP and ERAS should institute a properly organized second match during Match Week.
Mona M. Signer, executive director of the NRMP, disagrees with Roth's prediction that decision-making during the SOAP will be strategic. Instead, she predicts that programs and applicants will continue to pursue their best opportunities. Additionally, as with the Match, should a program or applicant violate prescribed rules, sanctions would be imposed.

My previous posts on the residency scramble, and the proposed new rules are here.

First year of the new medical residency scramble, SOAP

I've written before about the new Supplemental Offer and Acceptance Program (SOAP), and the National Resident Matching Program has now released a report on its first year of operation.

There were 1,100 unfilled first year positions at the end of the main match, and 815 unmatched seniors graduating from U.S. medical schools (and many more unmatched applicants when foreign medical schools are included). Most of the unmatched positions were in family medicine and in "preliminary" rotations in surgery and internal medicine.

After the first day of the SOAP exploding offer process (i.e. after two rounds of exploding offers), only 267 positions remained, and 98 of these remained unfilled. So, most of the action happened the first day.

Medical schools complained that students were asked to "commit" to programs prior to receiving an offer, and thought that rounds should be longer. Residency programs thought rounds should be shorter.

In line with the criticisms of the design offered earlier (see here), I anticipate that next year more students will be asked to "commit" before receiving an offer (even though it's against the rules), and that even more of the action will be concentrated in the first day and the first round, with more of the market shifting out of the formal scramble, either officially or de facto, through the offline "commitment" process....

As I was quoted saying last year (see here), "If it's really, really tempting for people on both sides to break the rules," says Roth, "often the rules get broken."


HT: Nikhil Agarwal

The job market in gastrointestinal endoscopy

After completing a 3 year subspecialty match in gastroenterology, doctors wishing to specialize further can do a fellowship in advanced endoscopy. The American Society for Gastrointestinal Endoscopy is trying to organize that job market, and, at least for this year, they are doing something quite different from a standard medical match. Aside from a system of prescribed dates (First date to offer an interview: 4/1/2011; First date to offer a position 7/15/2011: Fellowship start date: 7/1/2012), the process is described to applicants (in a letter) as follows:

******
"At 12pm EDT on July 15th, all program directors will send out an email to their top
choice. The fellow will then have 1 hour to decide if they wish to take that position or
wait for other offers. Please send a return email confirming that you got the offer.
You may respond at any time during that hour, ideally as soon as you make your
decision. If you do not respond within that hour, the program director may move on to
their second choice, so please respond within the hour.


"One of 2 things will then happen once you respond:


1. If you have chosen the offer, and send an affirmative email, the program
director will then send an email ASAP to all of its other applicants to
alert them that the spot has been filled, so that other applicants will be
aware that that position at that particular institution is no longer
available.


2. If you chose to reject the offer, please alert the program director via email
ASAP, so that the program director can then make an offer to the next
applicant on the list.


"If after the 15th (and the weekend of the 16th-17th) you do not have a position, please
go to the ASGE AEF website, and a list of programs with open positions will be
posted so that you may contact any of them if you like.


"I know that this non-electronic “match” is not ideal, but until we adopt an electronic
match (hopefully next year) we hope this format works without too many glitches."
******


Note that this is a system of "exploding offers", so one can expect some communication between participants before the appointed hour... (See also the discussion of similar problems I anticipate in the proposed new rules for the residency scramble (SOAP)).

Gastroenterology fellowships enjoy a successful match, so it seems reasonable to speculate that the fellowship in advanced endoscopy will turn to one after trying this.

Match Day for new doctors

Today is Match Day, when graduating medical students find out where they will do their residency.

MAR18

Match Day!

Applicants: Medical school Match Day ceremonies at 12:00 p.m. ET. Learn where you matched in the R3 system and by email at 1:00 p.m. ET.

Programs: Match Results by Ranked Applicant and SOAP Results by Preferred Applicant reports available at 2:00 p.m. ET.

Advance Data Tables available on www.nrmp.org at 1:00 p.m. ET.

Here's an earlier WSJ story by a graduating doctor:
Match Day, the NFL Draft of Medicine
On March 18, thousands of budding doctors find out where they’ll be serving their residencies.
"As you can imagine, it is a day of high anxiety, celebration and disappointment. My classmates and I have spent the past year choosing specialties, filling out applications and interviewing with residency programs. This process has taken us to cities, hospitals and universities around the country. After years of study, we’re about to become physicians. Both our professional and personal lives hinge on the results of this algorithm.

The Match was introduced in 1952 to address many of the uncertainties of residency applications. Before, medical students had to seek residency spots by themselves. But this decentralized system was rife with coercion, favoritism and deceit. Residency programs would demand medical students accept offers before applicants could consider other options. Students and programs betrayed one another by reneging on commitments.

The Match provides structure to the process. Students apply for residency positions through a common application. Strict policies govern communication between programs and applicants, with public reporting of violations. Applicants and programs submit ranking preferences into a centralized system. The Match algorithm couples these rankings and establishes binding contracts for residency positions.

Last year saw record numbers, when nearly 35,000 applicants submitted ranking preferences for roughly 30,000 residency slots. Medical schools host ceremonies to celebrate the event. Researchers Alvin Roth and Lloyd Shapley even won the 2012 Nobel Prize in Economics for their work related to the Match."
*****************
Here's the NRMP page on why it takes 23 days from the time preferences are submitted to announce the match results

What Happens Between Rank Order List Deadline And Match Day?

Between the Rank Order List Certification Deadline and Match Day, the NRMP conducts a rigorous review of Match data, including:

• Assessing and confirming the integrity of the data
• Conducting a final verification of applicants’ credentials
• Withdrawing applicants who are ineligible for the Match
• Transferring the data from the R3 system to the matching algorithm module, rechecking the data, processing the algorithm, and transferring data back into the R3 system
• Verifying the results of a Match and transferring the data into the NRMP databases
• Verifying applicants’ credentials for Main Residency Match SOAP participation
• Preparing 50,000 individual Match Week reports for Main Residency Match applicants, program directors, and medical schools

These steps, all completed in a few weeks, ensure the accuracy of Match results.

Here's a story on the SOAP scramble for unmatched students:https://www.statnews.com/2016/03/17/medical-students-match-day/ 

Transition from medical school to residency: defending the parts that work well (namely the NRMP Resident Match)

This post is about a recently published paper concerning the design of the market for new doctors in the U.S.  But it will require some background for most readers of this blog.   The short summary is that the market is experiencing problems related to congestion, and one of the proposals to address these problems was deeply flawed, and would have reduced market thickness and caused substantial direct harm to participants if implemented, and created instabilities that would likely have caused indirect harms to the match process in subsequent years. But this needed to be explained in the medical community, since that proposal was being  very actively advocated.

For those of you already steeped in the background, you can go straight to the paper, here.

Itai Ashlagi, Ephy Love, Jason I. Reminick, Alvin E. Roth; Early vs Single Match in the Transition to Residency: Analysis Using NRMP Data From 2014 to 2021. J Grad Med Educ 1 April 2023; 15 (2): 219–227. doi: https://doi.org/10.4300/JGME-D-22-00177.1

If the title doesn't remind you of the vigorous advocacy for an early match for select positions, here is some of the relevant back story.

The market for new doctors--i.e. the transition from medical school to residency--is experiencing growing pains as the number of applications and interviews has grown, which imposes costs on both applicants and residency programs.  

Below is a schematic of that process, which begins with applicants submitting applications electronically, which makes it easy to submit many.  This is followed by residency programs inviting some of their applicants to interview. The movement to Zoom interviews has made it easier to have many interviews also (although interviews were multiplying even before they moved to Zoom).  

After interviews, programs and applicants participate in the famous centralized clearinghouse called The Match, run by the NRMP. Programs and applicants each submit rank order lists (ROLs) ranking those with whom they interviewed, and a deferred acceptance algorithm (the Roth-Peranson algorithm) produces a stable matching, which is publicly announced on Match Day. (Unmatched people and positions are invited into a now computer-mediated scramble, called SOAP, and these matches too are announced on Match  Day.)

The Match had its origins as a way to control the "unraveling" of the market into inefficient bilateral contracts, in which employment contracts were made long before employment would commence, via exploding offers that left most applicants with very little ability to compare options.  This kind of market failure afflicted not only the market for new physicians (residents), but also the market for later specialization (as fellows). Consequently, over the years, many specialties have turned to matching for their fellowship positions as well.

  The boxes in brown in the schematic are those that constitute "The Match:" the formulation and submission of the ROLs, and the processing of these into a stable matching of programs to residents.  Congestion is bedeviling the parts in blue.

The boxes colored brown are 'The Match' in which participants formulate and submit rank order lists (ROLs), after which a deferred acceptance algorithm produces a stable matching of applicants to programs, which is accepted by programs and applicants on Match Day. The boxes in blue, the applications and interviews that precede the Match, are presently suffering from some congestion.  Some specialties have been experimenting with signals (loosely modeled on those in the market for new Economics PhDs, but implemented differently by different medical specialties).

The proposal in question was to divide the match into two matches, run sequentially, with the first match only allowing half of the available positions to be filled.  The particular proposal was to do this first for the OB-GYN specialty, thus separating that from the other specialties in an early match, with only half of the OB-GYN positions available early.

This proposal came out of a study funded by the American Medical Association, and it was claimed, without any evidence being offered, that it would solve the current problems facing the transition to residency.  Our paper was written to provide some evidence of the likely effects, by simulating the proposed process using the preferences (ROLs) submitted in previous years.  

The results show that the proposal would largely harm OB-GYN applicants by giving them less preferred positions than they could get in a traditional single match, and that it would create instabilities that would encourage strategic behavior that would likely undermine the successful operation of the match in subsequent years.

Itai Ashlagi, Ephy Love, Jason I. Reminick, Alvin E. Roth; Early vs Single Match in the Transition to Residency: Analysis Using NRMP Data From 2014 to 2021. J Grad Med Educ 1 April 2023; 15 (2): 219–227. doi: https://doi.org/10.4300/JGME-D-22-00177.1

Abstract:

"Background--An Early Result Acceptance Program (ERAP) has been proposed for obstetrics and gynecology (OB/GYN) to address challenges in the transition to residency. However, there are no available data-driven analyses on the effects of ERAP on the residency transition.

"Objective--We used National Resident Matching Program (NRMP) data to simulate the outcomes of ERAP and compare those to what occurred in the Match historically.

"Methods--We simulated ERAP outcomes in OB/GYN, using the de-identified applicant and program rank order lists from 2014 to 2021, and compared them to the actual NRMP Match outcomes. We report outcomes and sensitivity analyses and consider likely behavioral adaptations.

"Results--Fourteen percent of applicants receive a less preferred match under ERAP, while only 8% of applicants receive a more preferred match. Less preferred matches disproportionately affect DOs and international medical graduates (IMGs) compared to US MD seniors. Forty-one percent of programs fill with more preferred sets of applicants, while 24% fill with less preferred sets of applicants. Twelve percent of applicants and 52% of programs are in mutually dissatisfied applicant-program pairs (a pair in which both prefer each other to the match each received). Seventy percent of applicants who receive less preferred matches are part of a mutually dissatisfied pair. In 75% of programs with more preferred outcomes, at least one assigned applicant is part of a mutually dissatisfied pair.

"Conclusions--In this simulation, ERAP fills most OB/GYN positions, but many applicants and programs receive less preferred matches, and disparities increase for DOs and IMGs. ERAP creates mutually dissatisfied applicant-program pairs and problems for mixed-specialty couples, which provides incentives for gamesmanship."

************

I'm hopeful this paper will effectively contribute to the ongoing discussion of how, and how not, to modify the design of the whole process of transition to residency with an aim to fixing the parts that need fixing, without damaging the parts that work well, i.e. while doing no harm. 

(Signaling will likely continue to play a role in this.)