One of the fun things about our paper published in today's PNAS is that, as a working paper, it prompted a vigorous discussion of the ethics of doing field experiments in economics. We discuss this more fully in the published version, below:
J. Qiu, Y. Chen, A. Cohn, & A.E. Roth, Social media promotion improves job market outcomes, Proc. Natl. Acad. Sci. U.S.A. 123 (19) e2528289123, https://doi.org/10.1073/pnas.2528289123 (2026).
Abstract: Social media has transformed how academics disseminate research, but its effect on academic job outcomes remains unclear. Previous research has shown correlations between social media exposure and metrics like citation counts, but these relationships may be confounded by unobserved factors such as researcher quality or access to professional networks. We examine whether social media promotion causally affects job market outcomes in economics through a field experiment on Twitter (now X). We first collect tweets about job market papers from 519 candidates and post them from a dedicated account. We then randomize half of the posts to be quote-tweeted by established economists in the candidates’ fields, and measure the effects on both online visibility and hiring outcomes. We find that posts in the treatment group receive 441% more views and 303% more likes than those in the control group. Candidates whose posts were assigned to be quote-tweeted receive one additional flyout invitation compared to the control group average of 5.4 flyouts. Furthermore, women in the treatment group receive 0.9 more job offers than women in the control group, who receive 3 offers on average. Exploring mechanisms, we find that academic reputation drives these results, with stronger effects for quote-tweets from highly cited scholars and for candidates from top institutions. Our findings suggest social media promotion causally increases research visibility and improves academic job market outcomes.
...
"Ethical considerations.
"After the release of our working paper on the Social Science Research Network (SSRN) on May 20, 2024, a vigorous discussion arose on both social and mainstream media, particularly on Twitter, about the ethics of our experiment and of field experiments more generally (e.g., ref. 30). The main concern suggested that job markets are essentially constant sum, so that randomly promoting some candidates through having their JMPs quote-tweeted by influencers would necessarily (and unethically) disadvantage both those who were in the control condition of the experiment and those who did not participate in the experiment.
"We understand the importance of considering the ethical implications of any experiment and that ethicality is connected to the underlying economics of the job market. In this latter respect, given the information friction and congestion in the interview process, job markets are unlikely to be constant sum. Aside from the possibility of welfare gains from improved match quality, we note that, typically in matching markets, many employers fail to fill all their positions while at the same time qualified candidates fail to find one, so that welfare can also be improved by filling more positions. [In the 2022–2023 job market, the total number of jobs listed on JOE was 3,608, including 933 (1,083) full-time academic jobs in (outside) the United States and 718 full-time nonacademic jobs (any location). On the supply side, 1,386 Ph.D. students and postdocs applied to at least one job through JOE from August to December 2022 (31).] In economics, the job market often has unfilled positions by the end of February, leading to a scramble round each year starting in March. Similarly, the annual National Resident Matching Program (NRMP) for new physicians in the United States also leads to some positions being unfilled, despite having far more applicants than available positions. [For example, in 2024, 38,494 positions were offered to 44,853 active applicants and 2,510 positions were unfilled (6.5%), at the end of both the main match (a deferred acceptance algorithm, see ref. 32) and a centralized postmatch scramble called the Supplemental Offer and Acceptance Program (33).]
"The phenomenon of unfilled positions in a thick labor market may reflect congestion in the interview process. In such a market, since many positions receive more applications than the number of candidates who can feasibly be interviewed, the matching of interviews to jobs may be imperfect in the sense that an employer can find that none of the people interviewed can be successfully hired, but could have filled the position if more appropriate interviewees had been chosen. To mitigate this issue, signaling mechanisms have been introduced in both the economics and medical markets to facilitate a better matching of interviewees and employers (29, 34). In our context, the quote-tweeting of JMPs may similarly serve to help employers find better matches with their selection of interviewees who can be hired.
"We also propose that highlighting suitable candidates from underrepresented groups for a position could potentially expand the overall number of job openings. A notable example is the President’s Postdoctoral Fellowship Program, implemented across multiple institutions including the University of Michigan and the University of California system. This program seeks to recruit future faculty members “with the potential to bring to their research and undergraduate teaching the critical perspective that comes from their nontraditional educational background or understanding of the experiences of groups historically underrepresented in higher education.” (See, e.g., https://presidentspostdoc.umich.edu/, retrieved on August 29, 2025.)
"Finally, we consider trends in the broader context of job search in evaluating the ethical considerations related to our study. Social media has become a common channel for academics to advertise the JMPs of their students. Thus, we are not introducing a new channel for candidate promotion, nor are we excluding others outside of our experiment from availing themselves of this channel. Our goal is to understand the extent to which this channel may create visibility or improve outcomes for job candidates, especially since not all candidates may have equal access. Our paper belongs to the class of natural field experiment (35), a class that has seen a growing number of studies in which field experiments are used to assess the effects of market interventions. [A natural field experiment is one “where the subjects do not know that they are in an experiment” (35). In our context, participants were told only that we would arrange for their JMPs to be tweeted, but not that there would be a quote-tweet treatment.] One of the main benefits of conducting a natural field experiment is that it minimizes possible Hawthorne effects (36). These studies are widely accepted and even recognized, with the 2019 Nobel Prize for experiments in development economics. If it is ethical for economists to use experiments to evaluate interventions in other markets, it should also be ethical for economists to study the market for economists. And if it is ethical to promote students who are on the job market, then it should be ethical to study the effects of such promotion.
"In sum, from a normative perspective (should scholars promote candidates?), we argue that such promotion can reduce information friction and job market congestion, potentially leading to more efficient matching. From a positive perspective (does promotion matter?), we demonstrate in Results that it increases candidate visibility and improves job market outcomes, especially for women who are traditionally underrepresented in economics."
#############
Earlier (a blog post about reference 30, above):
Saturday, June 8, 2024 The ethics of field experiments in Economics, in the Financial Times
Posts
No comments:
Post a Comment