Enhancing Scientific Integrity in the Social and Behavioral Sciences--Nominate an Expert for an NAS workshop

 The National Academies of Sciences, Engineering, and Medicine are soliciting nominations for a workshop on

 Enhancing Scientific Integrity: Progress and Opportunities in the Social and Behavioral Sciences - A Workshop

"This workshop will bring together researchers, journal editors, publishers, funders, and scientific association leaders to identify practical, forward-looking strategies for strengthening data integrity and transparency in the social and behavioral sciences. Participants will explore innovative tools and frameworks to detect and prevent errors, promote accountability, and reinforce public trust in research. Discussions will also consider how journals, institutions, and professional societies can adopt fair, sustainable practices that support scientific rigor while ensuring accessibility for researchers across many contexts and settings.

Deadline: November 7, 2025

Call for Experts

We invite you to submit suggestions for experts to participate in this activity. The call for experts closes on November 7, 2025 at 11:59 PM PST.

 "The National Academies of Sciences, Engineering, and Medicine proposes to convene a workshop to bring together diverse stakeholders, including journal editors, publishers, scientists, funding agencies, and scientific association leaders, to advance research and data integrity. The workshop will focus on identifying proactive, constructive strategies to enhance transparency and accountability in research practices. Key questions to be addressed include:

• How can the social and behavioral sciences continue to lead the way in advancing data integrity? What successful methods or frameworks from other disciplines might be adapted to strengthen these efforts?

• Could systematic, random audits of published data help detect and correct honest mistakes while discouraging malfeasance? What governance structures would ensure such efforts are fair, sustainable, and constructive? What new tools might facilitate this process?

• How can scientific journals refine their policies (e.g., review processes, data validation) to support transparency and integrity while maintaining accessibility for researchers across diverse contexts?

• What strategies can be employed to ensure potential solutions avoid placing undue burdens on researchers, especially those at institutions with limited resources?"

Science and politics: Can you fix science by doing much less of it?

 Today's NYT has a long opinion piece about (my former Stanford colleague) Jay  Bhattacharya in his role as the new head of the National Institutes of Health (NIH).  I found it confusing and confused. But the last two paragraphs made some sense.

Jay Bhattacharya Wants to Fix Science.  Is He in Over His Head?  
By Ari Schulman 

Here are the two concluding paragraphs:

"The mRNA vaccine decision was the clearest test case yet of how his idealism will go once released into the wild. In The Washington Post, he acknowledges that the Covid vaccines saved millions of lives without known safety problems. He notes unanswered questions, around dosing and side effects. But when push came to shove, his response to these questions was not, let’s answer them with science, as he told me, but: Shut the science down. It turns out that the power of science to solve problems has limits after all.

"The problem with Dr. Bhattacharya is not that he’s cynical, as his critics say. It’s that his theory is naïve about power, and so could easily become a mouthpiece for it. America’s golden age of innovation, backed by levels of public investment that make us the envy of the world, has been nice while it’s lasted. If we want to keep it going, this moment may call less for a fresh infusion of reason than some new animating spirit, not a new Galileo but a new Robert Moses, Carl Sagan, or J. Robert Oppenheimer. Let us hope that Jay Bhattacharya still has it in him. The country needs it." 

The 2025 Golden Goose awards

 In these difficult times for science funding, the Golden Goose Award is a reminder of its benefits.

 Here's it's backstory. 

Here are the 2025 winners. 

“Nature has all the answers”

How a knack for nature’s oddities improved disease diagnostics & inspired scores of scientists

AWARDEE: Joseph G. Gall 

FEDERAL FUNDING AGENCIES: National Institutes of Health, National Science Foundation 

 and

Cisplatin Breakthrough Redefines Testicular Cancer Treatment 

 AWARDEES: Barnett Rosenberg, Loretta VanCamp, Thomas Krigas 

FEDERAL FUNDING AGENCIES: National Institutes of Health, National Science Foundation 

 ######### 

 Here are all my posts on the Golden Goose.  (Two of the early awards were for market design:)

Politics and science disagree about drinking and climate change...

 It turns our that worries about drinking were just as woke about worries about climate change.

The NYT has the story on drinking, CBS on climate change:

Federal Report on Drinking Is Withdrawn
The upcoming U.S. Dietary Guidelines will instead be influenced by a competing study, favored by industry, which found that moderate alcohol consumption was healthy. 

"The Department of Health and Human Services has pulled back a government report warning of a link between cancer and drinking even small amounts of alcohol, according to the authors of the research.

Their report, the Alcohol Intake and Health Study, warned that even one drink a day raises the risk of liver cirrhosis, oral and esophageal cancer, and injuries. The scientists who wrote it were told that the final version would not be submitted to Congress, as had been planned." 

##########

Here's CBS's story on climate change:

More than 85 climate experts say Energy Department report on greenhouse gases is "full of errors"

"An international group of more than 85 climate experts on Tuesday published a 439-page review arguing that a report by the Trump administration's Energy Department fails to "adequately represent the current scientific understanding of climate change," and it "exhibits pervasive problems" by misrepresenting scientific literature and cherry-picking data.

"The Department of Energy's 151-page report, "A Critical Review of Impacts of Greenhouse Gas Emissions on the U.S. Climate," was written by five authors who were hand-selected by Energy Secretary Chris Wright, a former fossil fuel executive. It included a controversial conclusion that "carbon dioxide-induced warming appears to be less damaging economically than commonly believed," and it states that "aggressive mitigation strategies" to address greenhouse gas emissions "could be more harmful than beneficial" — a statement that supports the oil and gas industry."

######

That has sparked a lawsuit by the Environmental Defense Fund.

Reputations take a long time to build but can be quickly destroyed

 US support for science is one of the things that will be hard to make reliable again.

Here's a story from the Washington Post, about and including an interview with the mathematician Terence Tao.

 The world’s greatest mathematician avoided politics. Then Trump cut science funding.  Terence Tao, often called the “Mozart of Math,” is focused on fundraising after federal research funding to UCLA was suspended.  By Carolyn Y. Johnson

" What’s hardest to restore is the sense of predictability and stability.

"People who support all the positive aspects of America have to speak out and fight for them now. The things that you took for granted, there was bipartisan support to keep certain things in the U.S. running as they have been more or less for the past 70 years because the system worked. That’s not a safe assumption anymore."

Fake science at scale, via organized paper mills (in PNAS and the NYT)

 Here's a recent paper from PNAS, analyzing the organized activity of paper mills producing fraudulent papers.

The entities enabling scientific fraud at scale are large, resilient, and growing rapidly  by Reese A. K. Richardson, Jennifer A. Byrne, and Luís A. Nunes Amaral
Edited by Daniel Acuña, University of Colorado Boulder, accepted by Editorial Board Member Mark Granovetter August 4, 2025  https://doi.org/10.1073/pnas.242009212

Abstract
Science is characterized by collaboration and cooperation, but also by uncertainty, competition, and inequality. While there has always been some concern that these pressures may compel some to defect from the scientific research ethos—i.e., fail to make genuine contributions to the production of knowledge or to the training of an expert workforce—the focus has largely been on the actions of lone individuals. Recently, however, reports of coordinated scientific fraud activities have increased. Some suggest that the ease of communication provided by the internet and open-access publishing have created the conditions for the emergence of entities—paper mills (i.e., sellers of mass-produced low quality and fabricated research), brokers (i.e., conduits between producers and publishers of fraudulent research), predatory journals, who do not conduct any quality controls on submissions—that facilitate systematic scientific fraud. Here, we demonstrate through case studies that i) individuals have cooperated to publish papers that were eventually retracted in a number of journals, ii) brokers have enabled publication in targeted journals at scale, and iii), within a field of science, not all subfields are equally targeted for scientific fraud. Our results reveal some of the strategies that enable the entities promoting scientific fraud to evade interventions. Our final analysis suggests that this ability to evade interventions is enabling the number of fraudulent publications to grow at a rate far outpacing that of legitimate science.

#########

And here's the story in the NYT:

Fraudulent Scientific Papers Are Rapidly Increasing, Study Finds
A statistical analysis found that the number of fake journal articles being churned out by “paper mills” is doubling every year and a half. by Carl Zimmer

 

Open Letter from NAS Members to U.S. Senators and Representatives urging support for science

In a victory of optimism over experience, I continue to sign a small fraction of the open letters that come my way.  Also, it's good to exercise Americans' right to petition the government for the redress of grievances.

Below is a link to a letter organized by Professor Walter Leal with well over 1,000 signatures from members of the National Academy of Sciences. (Organizing NAS members to petition the government isn't so easy, since it has to be done without the help of the NAS, which as a government-related organization is being quite cautious in these times.)

Letter to Congress Requesting Protection of Science Funding in the FY26 Budget

Or, if you prefer to hear the letter read out loud, that takes about three minutes on this YouTube video (which also shows the text, but not the signers).

 

#############

Here's another take on more or less the same subject, from the Chronicle of Higher Education:

 The Wrecking of American Research Federal policy made universities in the United States the envy of the world. One presidency could end that.
Broken Promise  By Karin Fischer

" as the United States approaches its 250th birthday, the country’s research edifice is in danger of collapse, battered by a wrecking ball known as the Trump administration."

Stanford celebrates the NSF–DOE Vera C. Rubin Observatory (and recalls university-government collaboration on science)

 Remember when universities and the Federal government collaborated on big science?

The bold bet that built a telescope

"When the first images from the NSF–DOE Vera C. Rubin Observatory were released on June 23, they marked a historic milestone for the Legacy Survey of Space and Time (LSST), a landmark 10-year campaign to map the southern sky with the world’s largest digital camera, set to begin full science operations later this year. 

"Today, Rubin is an $800 million observatory backed by the National Science Foundation (NSF) and the Department of Energy (DOE). But two decades ago, it was little more than a vision without funding, a home, or agency support.

"That changed in 2003, when Stanford University and SLAC National Accelerator Laboratory jointly launched the Kavli Institute for Particle Astrophysics and Cosmology (KIPAC), setting in motion a chain of events that helped bring the LSST to life." 

 

the digital camera...

Interdisciplinary science: a heated dispute

 Here's an article in the latest PNAS, about issues in evolution that I know nothing about, but I was struck by how clearly the author makes plain in the abstract his view that some authors of other papers also know nothing.

Complexity myths and the misappropriation of evolutionary theory  by Michael Lynch, Edited by Nils Stenseth    https://doi.org/10.1073/pnas.2425772122

Abstract: Recent papers by physicists, chemists, and geologists lay claim to the discovery of new principles of evolution that have somehow eluded over a century of work by evolutionary biologists, going so far as to elevate their ideas to the same stature as the fundamental laws of physics. These claims have been made in the apparent absence of any awareness of the theoretical framework of evolutionary biology that has existed for decades. The numerical indices being promoted suffer from numerous conceptual and quantitative problems, to the point of being devoid of meaning, with the authors even failing to recognize the distinction between mutation and selection. Moreover, the promulgators of these new laws base their arguments on the idea that natural selection is in relentless pursuit of increasing organismal complexity, despite the absence of any evidence in support of this and plenty pointing in the opposite direction. Evolutionary biology embraces interdisciplinary thinking, but there is no fundamental reason why the field of evolution should be subject to levels of unsubstantiated speculation that would be unacceptable in any other area of science.

NSF slashed again

The "S" in NSF has again attracted the attention of the Trump administration.

 The journal Science reports the story: 

 NSF faces radical shake-up as officials abolish its 37 divisions
Changes seen as a response to presidential directives on what research to fund By Jeffrey Mervis

"The National Science Foundation (NSF), already battered by White House directives and staff reductions, is plunging into deeper turmoil. According to sources who requested anonymity for fear of retribution, staff were told today that the agency’s 37 divisions—across all eight NSF directorates—are being abolished and the number of programs within those divisions will be drastically reduced. The current directors and deputy directors will lose their titles and might be reassigned to other positions at the agency or elsewhere in the federal government.

The consolidation appears to be driven in part by President Donald Trump’s proposal to cut the agency’s $4 billion budget by 55% for the 2026 fiscal year that begins on 1 October. NSF’s decision to abolish its divisions could also be part of a larger restructuring of the agency’s grantmaking process that involves adding a new layer of review. NSF watchers fear that a smaller, restructured agency could be more vulnerable to pressure from the White House to fund research that suits its ideological bent.

As soon as this evening, NSF is also expected to send layoff notices to an unspecified number of its 1700-member staff. ... The agency is also expected to issue another round of notices tomorrow terminating grants that have already been awarded, sources say. In the past 3 weeks, the agency has pulled the plug on almost 1400 grants worth more than $1 billion."

CDC’s laboratory on sexually transmitted diseases is shut by Trump administration

 Statnews has the story:

CDC’s top laboratory on sexually transmitted diseases is shut by Trump administration
‘We are blind,’ researcher says, noting the lab is crucial to tracking drug-resistant gonorrhea and other diseases  By Helen Branswell April 5, 2025

"At a time when the world is down to a single drug that can reliably cure gonorrhea, the U.S. government has shuttered the country’s premier sexually transmitted diseases laboratory, leaving experts aghast and fearful about what lies ahead.

"The STD lab at the Centers for Disease Control and Prevention — a leading player in global efforts to monitor for drug resistance in the bacteria that cause these diseases — was among the targets of major staff slashing at the CDC this past week. All 28 full-time employees of the lab were fired.

...

"A CDC white paper on antibiotic resistance released during the first Trump administration listed drug-resistant gonorrhea as one of five urgent threats facing the country. Antimicrobial resistance to that last drug that reliably works to cure gonorrhea, ceftriaxone, is rare but on the rise globally."

 

Open letter on Science

 Here's the NYT story out this morning,

Trump Administration Has Begun a War on Science, Researchers. Say Nearly 2,000 scientists urged that Congress restore funding to federal agencies decimated by recent cuts.

 

And here's the letter (and all the signatures):
TO THE AMERICAN PEOPLE
We all rely on science. Science gave us the smartphones in our pockets, the navigation systems in our cars, and life-saving medical care. We count on engineers when we drive across bridges and fly in airplanes. Businesses and farmers rely on science and engineering for product innovation, technological advances, and weather forecasting. Science helps humanity protect the planet and keeps pollutants and toxins out of our air, water, and food.

For over 80 years, wise investments by the US government have built up the nation’s research enterprise, making it the envy of the world. Astoundingly, the Trump administration is destabilizing this enterprise by gutting funding for research, firing thousands of scientists, removing public access to scientific data, and pressuring researchers to alter or abandon their work on ideological grounds.

The undersigned are elected members of the National Academies of Sciences, Engineering, and Medicine, representing some of the nation’s top scientists, engineers, and medical researchers. We are speaking out as individuals. We see real danger in this moment. We hold diverse political beliefs, but we are united as researchers in wanting to protect independent scientific inquiry. We are sending this SOS to sound a clear warning: the nation’s scientific enterprise is being decimated.

The administration is slashing funding for scientific agencies, terminating grants to scientists, defunding their laboratories, and hampering international scientific collaboration. The funding cuts are forcing institutions to pause research (including studies of new disease treatments), dismiss faculty, and stop enrolling graduate students—the pipeline for the next generation’s scientists.

The administration’s current investigations of more than 50 universities send a chilling message. Columbia University was recently notified that its federal funding would be withheld unless it adopted disciplinary policies and disabled an academic department targeted by the administration. Destabilizing dozens of universities will endanger higher education—and the research those institutions conduct.

The quest for truth—the mission of science—requires that scientists freely explore new questions and report their findings honestly, independent of special interests. The administration is engaging in censorship, destroying this independence.  It is using executive orders and financial threats to manipulate which studies are funded or published, how results are reported, and which data and research findings the public can access. The administration is blocking research on topics it finds objectionable, such as climate change, or that yields results it does not like, on topics ranging from vaccine safety to economic trends.

A climate of fear has descended on the research community. Researchers, afraid of losing their funding or job security, are removing their names from publications, abandoning studies, and rewriting grant proposals and papers to remove scientifically accurate terms (such as “climate change”) that agencies are flagging as objectionable. Although some in the scientific community have protested vocally, most researchers, universities, research institutions, and professional organizations have kept silent to avoid antagonizing the administration and jeopardizing their funding.

If our country’s research enterprise is dismantled, we will lose our scientific edge. Other countries will lead the development of novel disease treatments, clean energy sources, and the new technologies of the future. Their populations will be healthier, and their economies will surpass us in business, defense, intelligence gathering, and monitoring our planet’s health. The damage to our nation’s scientific enterprise could take decades to reverse.

We call on the administration to cease its wholesale assault on U.S. science, and we urge the public to join this call. Share this statement with others, contact your representatives in Congress, and help your community understand what is at risk. The voice of science must not be silenced.  We all benefit from science, and we all stand to lose if the nation’s research enterprise is destroyed.  

The views expressed here are our own and not those of the National Academies or our home institutions.

Demonstrating for science (in Washington and elsewhere)

 Scientists are more accustomed to demonstrating science than demonstrating for science, but that may need to change.

Nature has the story:

NEWS, 03 March 2025
US science is under threat ― now scientists are fighting back
Researchers are organizing protests and making their voices heard as Trump officials slash funding and lay off federal scientists. By Heidi Ledford & Alexandra Witze 

"Across the United States, researchers are navigating uncomfortable territory. Repeated threats to research funding and the mass firings of federal workers have pushed some scientists to take on unfamiliar roles as activists, speaking at rallies, calling legislators and forming new pressure groups. “Historically, scientists have done a really bad job of advocating for their own activities,” says David Meyer, a sociologist at the University of California, Irvine. “So this is a new challenge.”

Unaccustomed role

The events of the past six weeks have compelled many scientists to embrace that challenge. Soon after the second inauguration of US President Donald Trump on 20 January, the new administration attempted to freeze payments on federal grants; announced that it would review and potentially cancel any grant that mentioned terms it deemed indicative of diversity, equity and inclusion (DEI) programmes; and issued dramatic cuts to the overhead, or ‘indirect costs’, paid on projects funded by the US National Institutes of Health.

...

"For many scientists, the big event is coming up on 7 March, at ‘Stand Up for Science’ rallies slated to take place in 32 cities around the country. The main event, in Washington DC, is spearheaded by a group of five researchers, most of them graduate students, who came together to combat their own initial feelings of powerlessness. “It’s been inspiring, as this has grown, to see how many people were feeling the same way and to take action,” says Emma Courtney, a graduate student in biology at Cold Spring Harbor Laboratory in New York."

National Medals of Science and Technology (including Cynthia Dwork for differential privacy)

 In one of the final acts of his administration, President Biden celebrates 25 distinguished scientists and engineers. (I'm particularly glad to see Cynthia Dwork recognized for her work on differential privacy.)

 Forbes has the story:

Biden Names 25 Recipients Of National Medals Of Science, Technology, by Michael T. Nietzel

In a statement from the White House, Biden said, “those who earn these awards embody the promise of America by pushing the boundaries of what is possible. These trailblazers have harnessed the power of science and technology to tackle challenging problems and deliver innovative solutions for Americans and for communities around the world.”

...

"The 14 recipients of the National Medal of Science are:

    Richard B. Alley, the Evan Pugh University Professor of Geosciences at Pennsylvania State University. Alley researches the great ice sheets to help predict future changes in climate and sea levels.
    Larry Martin Bartels, University Distinguished Professor of Political Science and Law and the May Werthan Shayne Chair of Public Policy and Social Science at Vanderbilt University. His scholarship focuses on public opinion, public policy, election science, and political economy.
    Bonnie L. Bassler, Squibb Professor in Molecular Biology and chair of the Department of Molecular Biology at Princeton University, for her research on the molecular mechanisms that bacteria use for intercellular communication.
    Angela Marie Belcher, the James Mason Crafts Professor of Biological Engineering and Materials Science and Engineering at MIT and a member of the Koch Institute for Integrative Cancer Research. She was honored for designing materials for applications in solar cells, batteries, and medical imaging.
    Helen M. Blau, Donald E. and Delia B. Baxter Foundation Professor and the Director of the Baxter Laboratory for Stem Cell Biology at Stanford University for her research on muscle diseases, regeneration and aging, including the use of stem cells for tissue repair.
    Emery Neal Brown, the Edward Hood Taplin Professor of Medical Engineering and Computational Neuroscience at MIT, was recognized for his work revealing how anesthesia affects the brain.
    John O. Dabiri, Centennial Chair Professor at the California Institute of Technology, in the Graduate Aerospace Laboratories and Mechanical Engineering. His research focuses on fluid mechanics and flow physics, with an emphasis on topics relevant to biology, energy, and the environment.
    Ingrid Daubechies, the James B. Duke Distinguished Professor Emerita of Mathematics at Duke University, was honored for her pioneering work on signal processing.
    Cynthia Dwork, Gordon McKay Professor of Computer Science at Harvard University, was recognized for research that has transformed the way data privacy is handled in the age of big data and AI.
    R. Lawrence Edwards, Regents and Distinguished McKnight University Professor, Department of Earth and Environmental Sciences at the University of Minnesota. Edwards is known for his refinement of radiocarbon dating techniques to study climate history and ocean chemistry.
    Wendy L. Freedman, the John and Marion Sullivan University Professor in Astronomy and Astrophysics at the University of Chicago, for her observational cosmology research, including pioneering uses of the Hubble Space Telescope.
    Keivan G. Stassun, Stevenson Professor of Physics & Astronomy at Vanderbilt University for his work in astrophysics, including the study of star formation and exoplanets.
    G. David Tilman is Regents Professor and the McKnight Presidential Chair in Ecology, Evolution, and Behavior at the University of Minnesota. He studies biological diversity, the structure and benefits of ecosystems and ways to assure sustainability despite global increases in human consumption and population.
    Teresa Kaye Woodruff is the MSU Research Foundation Professor of Obstetrics, Gynecology and Reproductive Biology and Biomedical Engineering at Michigan State University. She is an internationally recognized expert in ovarian biology and reproductive science.

The nine individual recipients of the National Medal of Technology and Innovation are:

    Martin Cooper for his work in advancing in personal wireless communications for over 50 years. Cited in the Guinness Book of World Records for making the first cellular telephone call, Cooper, known as the “father of the cell phone,” spent much of his career at Motorola.
    Jennifer A. Doudna, a Nobel Laureate in Chemistry and the Li Ka Shing Chancellor’s Chair in Biomedical and Health Sciences at the University of California, Berkeley. She is a pioneer of CRISPR gene editing.
    Eric R. Fossum is the John H. Krehbiel Sr. Professor for Emerging Technologies at Dartmouth College. He invented the CMOS active pixel image sensor used in cell-phone cameras, webcams, and medical imaging.
    Paula T. Hammond, an MIT Institute Professor, vice provost for faculty, and member of the Koch Institute, was honored for developing methods for assembling thin films that can be used for drug delivery, wound healing, and other applications.
    Kristina M. Johnson, former president of The Ohio State University was recognized for research in photonics, nanotechnology, and optoelectronics. Her discoveries have contributed to sustainable energy solutions and advanced manufacturing technologies.
    Victor B. Lawrence spent much of his career at Bell Laboratories, working on new developments in multiple forms of communications. He is a Research Professor and Director of the Center for Intelligent Networked Systems at Stevens Institute of Technology.
    David R. Walt is a faculty member of the Wyss Institute at Harvard University and is the Hansjörg Wyss Professor of Bioinspired Engineering at Harvard Medical School and Professor of Pathology at Harvard Medical School and Brigham and Women’s Hospital. He was honored for co-inventing the DNA microarray, enabling large-scale genetic analysis and better personalized medicine.
    Paul G. Yock is an emeritus faculty member at Stanford University. A physician, Yock is known for inventing, developing and testing new cardiovascular intervention devices, including the stent.
    Feng Zhang, the James and Patricia Poitras Professor of Neuroscience at MIT and a professor of brain and cognitive sciences and biological engineering, was recognized for his work developing molecular tools, including the CRISPR genome-editing system."

#########

Here's my post from ten years ago:

Saturday, February 7, 2015 Differential Privacy: an appreciation of Cynthia Dwork

 

An own-goal in replication science--retraction of a paper that reported high replicability

  A 2023 paper reporting high replicability of psychology experiments has been retracted from Nature Human Behavior. The retraction notice says in part 
"The concerns relate to lack of transparency and misstatement of the hypotheses and predictions the reported meta-study was designed to test; lack of preregistration for measures and analyses supporting the titular claim (against statements asserting preregistration in the published article); selection of outcome measures and analyses with knowledge of the data; and incomplete reporting of data and analyses."

RETRACTED ARTICLE: High replicability of newly discovered social-behavioural findings is achievable

• John Protzko, 
• Jon Krosnick, 
• Leif Nelson, 
• Brian A. Nosek, 
• Jordan Axt, 
• Matt Berent, 
• Nicholas Buttrick, 
• Matthew DeBell, 
• Charles R. Ebersole, 
• Sebastian Lundmark, 
• Bo MacInnis, 
• Michael O’Donnell, 
• Hannah Perfecto, 
• James E. Pustejovsky, 
• Scott S. Roeder, 
• Jan Walleczek & 
• Jonathan W. Schooler 

Nature Human Behaviour volume 8, pages311–319 (2024)Cite this article

• 47k Accesses

• 470 Altmetric

• Metricsdetails

This article was retracted on 24 September 2024

Matters Arising to this article was published on 24 September 2024

This article has been updated

Abstract

Failures to replicate evidence of new discoveries have forced scientists to ask whether this unreliability is due to suboptimal implementation of methods or whether presumptively optimal methods are not, in fact, optimal. This paper reports an investigation by four coordinated laboratories of the prospective replicability of 16 novel experimental findings using rigour-enhancing practices: confirmatory tests, large sample sizes, preregistration and methodological transparency. In contrast to past systematic replication efforts that reported replication rates averaging 50%, replication attempts here produced the expected effects with significance testing (P < 0.05) in 86% of attempts, slightly exceeding the maximum expected replicability based on observed effect sizes and sample sizes. When one lab attempted to replicate an effect discovered by another lab, the effect size in the replications was 97% that in the original study. This high replication rate justifies confidence in rigour-enhancing methods to increase the replicability of new discoveries.

########

############

In general, I'm more optimistic about replications than preregistrations for identifying replicable results and testing hypotheses about them.  In this case, preregistration apparently revealed that what was written up as a replication study had begun as something else, and that the goal posts had been moved ex post, apparently in inappropriate ways.

######

Somewhat related are my posts on the Einstein Foundation Award for Promoting Quality in Research.

The NAS proposes that bans on studying marijuana and its effects should be relaxed

 The National Academy of Sciences has just issued a new report on marijuana and public health.  Among their recommendations is that bans on research should be rescinded. (Because marijuana is currently a Schedule I drug in the Controlled Substances Act, it's hard to get permission to study it and its effects...)

Cannabis Policy Impacts Public Health and Health Equity (2024)

Golden Goose awards for woodpeckers, penguins, and artificial intelligence

 The Golden Goose awards are given each year to "recognize the tremendous human and economic benefits of federally funded research by highlighting examples of seemingly obscure studies that have led to major breakthroughs and resulted in significant societal impact."

This year they recognize three streams of work, that have led to the recovery of an endangered woodpecker species, to the more effective counting of penguins, and to the invention of neural nets on which the current artificial intelligence industry is based.

Here are those stories.

It’s a Family Affair: The Resurgence of the Red-Cockaded Woodpecker  AWARDEE: Jeff Walters

From Poop to Protection: Satellite Discoveries Help Save Antarctic Penguins and Advance Wildlife Monitoring  AWARDEES: Christian Che-Castaldo, Heather Joan Lynch, Mathew Schwaller

How We Think: Brain-Inspired Models of Human Cognition Contribute to the Foundations of Today’s Artificial Intelligence  AWARDEES: Geoffrey Hinton, James L. McClelland, David E. Rumelhart

Here's the first paragraph of the description of this third award (last but not least:)

"Decades before artificial intelligence emerged as the platform for innovation that it is today, David Rumelhart, James McClelland, and Geoffrey Hinton were exploring a new model to explain human cognition. Dissatisfied with the prevailing symbolic theory of cognition, David Rumelhart began to articulate the need for a new approach to modeling cognition in the mid-1970s, teaming up with McClelland with support from the National Science Foundation to create a model of human perception that employed a new set of foundational ideas. At around the same time, Don Norman, an early leader in the field of cognitive science, obtained funding from the Sloan Foundation to bring together an interdisciplinary group of junior scientists, including Hinton, with backgrounds in computer science, physics, and neuroscience. Rumelhart, McClelland, and Hinton led the development of the parallel distributed processing framework, also known as PDP, in the early-1980s, focusing on how networks of simple processing units, inspired by the properties of neurons in the brain, could give rise to human cognitive abilities. While many had dismissed the use of neural networks as a basis for building models of cognition in the 1960s and 1970s, the PDP group revived interest in the approach. Skeptics critiqued the new models too, and had only limited success in enabling effective artificially intelligent systems until the 2010s, when massive increases in the amount of available data and computer power enabled Hinton and others to achieve breakthroughs leading to an explosion of new technological advancements and applications."

############

Prior awards included market design in 2013 and 2014.

A 40 year old proof about top trading cycles is corrected (by two Berkeley grad students)

 Science (and math) can be self-correcting, sometimes slowly.  Here's an article that corrects the first proof that the top trading cycles algorithm is group strategy proof.  That's a true result, with multiple subsequent proofs.  But apparently the first proof presented wasn't the best one.  That's good to know.

One reason this may have taken a long time to spot is that the result is correct, and that there are subsequent proofs that connect the result to properties of other mechanisms.  

Will Sandholtz and Andrew Tai, the authors, did this work as Ph.D. students at UC Berkeley. (good for them!)

Group incentive compatibility in a market with indivisible goods: A comment  by Will Sandholtz and Andrew Tai

"Highlights

• Bird (1984), first to show top trading cycles is group strategy-proof, has errors.

•We present corrected results and proofs.

•We present a novel proof of strong group strategy-proofness without non-bossiness.

"Abstract: We note that the proofs of Bird (1984), the first to show group strategy-proofness of top trading cycles (TTC), require correction. We provide a counterexample to a critical claim and present corrected proofs in the spirit of the originals. We also present a novel proof of strong group strategy-proofness using the corrected results."

Fraud in physics? Room temp superconductors, again

 It should come as no surprise that it's not only social sciences that can be roiled by accusations of research misconduct.

Here's a story in Nature about a scientist who had a paper retracted from Nature, and then had another accepted, and then also retracted, both about room temperature superconductors.  It's a long, detailed story, but it says something about both science and about peer review.

Superconductivity scandal: the inside story of deception in a rising star’s physics lab. Ranga Dias claimed to have discovered the first room-temperature superconductors, but the work was later retracted. An investigation by Nature’s news team reveals new details about what happened — and how institutions missed red flags.   By Dan Garisto

"A researcher at the University of Rochester in New York, Dias achieved widespread recognition for his claim to have discovered the first room-temperature superconductor, a material that conducts electricity without resistance at ambient temperatures. Dias published that finding in a landmark Nature paper1.

"Nearly two years later, that paper was retracted. But not long after, Dias announced an even bigger result, also published in Nature: another room-temperature superconductor2.

...

" Nature has since retracted his second paper2 and many other research groups have tried and failed to replicate Dias’s superconductivity results. ...The scandal “has damaged careers of young scientists...

...

"Three previous investigations ... by the University of Rochester did not find evidence of misconduct. But last summer, the university launched a fourth investigation,... That fourth investigation is now complete and, according to a university spokesperson, the external experts confirmed that there were “data reliability concerns” 

...

"Nature retracted the CSH paper on 26 September 2022, with a notice that states “issues undermine confidence in the published magnetic susceptibility data as a whole, and we are accordingly retracting the paper”.

...

"Felicitas Heβelmann, a specialist in retractions at the Humboldt University of Berlin, says misconduct is difficult to prove, so journals often avoid laying blame on authors in retractions. “A lot of retractions use very vague language,” she says.

...

"The lack of industry-wide standards for investigating misconduct leaves it unclear whether the responsibility to investigate lands more on journals or on institutions.

...

"After Nature published the LuH paper in March 2023, many scientists were critical of the journal’s decision, given the rumours of misconduct surrounding the retracted CSH paper.

...

"All four referees agreed that the findings, if true, were highly significant. But they emphasized caution in accepting the manuscript, because of the extraordinary nature of the claims. Referee 4 wrote that the journal should be careful with such extraordinary claims to avoid another “Schön affair”, referring to the extensive data fabrication by German physicist Jan Hendrik Schön, which has become a cautionary tale in physics and led to dozens of papers being retracted, seven of them in Nature. Referees 2 and 3 also expressed concern about the results because of the CSH paper, which at the time bore an editor’s note of concern but had not yet been retracted. 

...

"When asked why Nature considered Dias’s LuH paper after being warned of potential misconduct on the previous paper, Magdalena Skipper, Nature’s editor-in-chief, said: “Our editorial policy considers every submission in its own right.” The rationale, Skipper explains, is that decisions should be made on the basis of the scientific quality, not who the authors are."