Modern peer review: how it evolved since the 1950's in Sociology

 Peer review is a big part of the design of modern academic publishing in scholarly journals. It wasn't always that way, and the current peer review system is pretty modern. Here's an account of its development in the discipline of Sociology, since the 1950's (which is similar to what we see in Economics, except that it appears Sociology relies substantially more on double-blind reviews).

Merriman, B. Peer Review as an Evolving Response to Organizational Constraint: Evidence from Sociology Journals, 1952–2018. The American Sociologist 52, 341–366 (2021). https://doi.org/10.1007/s12108-020-09473-x

"Abstract: Double-blind peer review is a central feature of the editorial model of most journals in sociology and neighboring social scientific fields, yet there is little history of how and when its main features developed. Drawing from nearly 70 years of annual reports of the editors of American Sociological Association journals, this article describes the historical emergence of major elements of editorial peer review. These reports and associated descriptive statistics are used to show that blind review, ad hoc review, the formal requirement of exclusive submission, routine use of the revise and resubmit decision, and common use of desk rejection developed separately over a period of decades. The article then argues that the ongoing evolution of the review model has not been driven by intellectual considerations. Rather, the evolution of peer review is best understood as the product of continuous efforts to steward editors’ scarce attention while preserving an open submission policy that favors authors’ interests."

From the introduction:

"In the main, editors are faculty members who operate a journal concurrently with ordinary work responsibilities; some receive modest, fixed remuneration, but editors have no strong financial interest in the journals they edit, and commonly serve for fixed or periodically renewed terms. (For most current journals, the economic interest rests primarily with one of a handful of large commercial publishers.) Journals do not restrict submissions by status criteria such as institutional affiliation or academic rank, and submission ordinarily carries little or no money cost, even at journals where authors assume a large part of the eventual expense of publication. Authors are expected to submit a given work exclusively to a single journal. After initial screening, submissions to a typical journal undergo double-blind review, in which the identities of authors and reviewers are not known to one another. Most evaluations of submitted manuscripts are produced by scholars who are not part of the appointed editorial staff of the journal. Work that is published has ordinarily undergone at least one formal round of revision and resubmission in response to the substance of external evaluations.

...

"At ASA journals, blind review, external review, exclusive submission, the formal revise and resubmit decision, and a developmental (rather than advisory) model of assessment developed in succession over a period of more than 30 years. In the twenty-first Century, persistent difficulties in obtaining timely reviews prompted a rapid, order of magnitude increase in frequency of rejections without review, commonly called desk rejections. Blind review was the only feature of the present model adopted at an ASA journal with an explicitly stated, unambiguously intellectual aim. This article argues that the other features of the current peer review model emerged as improvised efforts to balance two competing organizational imperatives: editors must steward scarce time and attention, but have also sought to render reasonably timely decisions without a priori exclusion of large numbers of prospective authors or capricious rejection of submissions. This pattern in journal operations in many ways reflects larger structural changes in sociology: rapid expansion of the field in the mid-to-late twentieth Century was succeeded by increasing competition in the academic labor market and heightened publishing expectations for tenure and promotion."

A snippet from the body of the paper:

"Ad hoc review, in which manuscripts are referred to scholars not formally affiliated with the journal organization, did not become an integral feature of the ordinary reviewing process until the early 1970s. Before then, virtually all evaluations were produced by members of journal editorial boards. Exclusive submission to a single journal also did not become a rule until the 1970s, and there is suggestive evidence that simultaneous submission to multiple journals may have been somewhat common until that time.

"At first, reviewing was plainly intended to aid to the work of the editor; the occasional value of reviews to authors was taken as an incidental benefit. The evolution of a developmental model of review oriented toward the author was gradual, as was the emergence of the revise and resubmit decision as a nearly unavoidable intermediate step on the path to publication."

...

And in conclusion:

"A primary constraint on editorial innovation is, of course, the professional and status structure of academic disciplines. An extensive body of research on disciplines, and on higher educational institutions more generally, has shown a powerful isomorphic tendency: such structures tend to converge on a given form of practice even if all the actors are wholly aware of its inadequacies. Further, change in such practices will, under most circumstances, be slow: individual academic advancement involves regularly submitting oneself to the judgment of the more experienced members of a discipline according to the standards those more experienced scholars impose. Those who may have the freshest view of an intellectual field, and perhaps a greater impulse to explore new lines of work, also face the strongest pressures to invest their time and effort conservatively in the oldest means of publicizing their work.

"Efforts to change publishing norms therefore stand a much greater chance of success if they are adopted first, or early, by actors who occupy central places in a field, or if they are given the strong, credible endorsement of such actors (Starbuck 2016: 178). Conversations about academic publishing models, especially their relative unresponsiveness to changing circumstances in the twenty-first Century, often possess a degree of fatalism. But the development of editorial peer review itself is an important reminder of how rapidly a good idea may spread."

HT: Retraction Watch