Limiting job applications in an online labor market: by John Horton and Shoshana Vasserman

 Here's an experiment that involved limiting the number of applications to particular jobs in an online labor market, in which many applicants are likely close substitutes.

Job-Seekers Send Too Many Applications: Experimental Evidence and a Partial Solution by John J. Horton and Shoshana Vasserman

Abstract: As job-seekers internalize neither the full benefits or costs of their application decisions, job openings do not necessarily obtain the socially efficient number of applications. Using a field experiment conducted in an online labor market, we find that some job openings receive far too many applications, but that a simple intervention can improve the situation. A treated group of job openings faced a soft cap on applicant counts. However, employers could easily opt out by literally clicking a single button. This tiny imposed cost on the demand side had large effects on the supply side, reducing the number of applicants to treated jobs by 11%—with even larger reductions in jobs where additional applicants were likely to be inframarginal. This reduction in applicant counts had no discernible effect on the probability a hire was made, or in the quality of the subsequent match. This kind of intervention is easy to implement by any online marketplace or job board and has attractive properties, saving job-seekers effort while still allowing employers with high marginal returns to more applicants to get them.

"In this paper, we describe an experiment conducted in an online labor market that influenced the size of applicant pools faced by employers.1 This was done by imposing a soft cap on the number of applicants that a job opening could receive, as well as limiting the duration of the window of time during which applications could be received: when a job opening received 50 applicants—or when 120 hours (5 days) had passed—no more applicants could apply unless the employer explicitly asked for more applicants. The intent of the intervention was to prevent job-seekers from applying to jobs where their application was likely to either be ignored or simply displace some other applicant, without preventing employers with high marginal returns to more applicants from obtaining them.

...

There is no evidence that better or worse matches were made in the treatment group, as measured by the feedback given by the employer at the end of the contract or in hours-worked. If anything, employer satisfaction rose slightly in the treatment.

The lack of effects on hiring or match quality is seemingly surprising, but likely reflects the fact that price competition among workers “prices in” vertical differences among workers, leaving firms close to indifferent over applicants, as in Romer (1992). Because of this indifference, substitution among applicants is not very costly to employers.

...

"only about 7% of employers requested more applicants by pushing the button.

"The treatment intervention likely saved job-seekers substantial time—more so than the percentage changes in job post applicant counts would seemingly imply. To see why the treatment has out-sized effects on job seekers, note that although relatively few job openings were affected by the 50 applicant cap (about 10%), these job openings are disproportionately important to job-seekers, as they attracted 43% of applications. This difference simply reflects the fact that a randomly selected application is more likely to be sent to a job with a high applicant count.