Showing posts with label science. Show all posts
Showing posts with label science. Show all posts

Tuesday, October 8, 2024

An own-goal in replication science--retraction of a paper that reported high replicability

  A 2023 paper reporting high replicability of psychology experiments has been retracted from Nature Human Behavior. The retraction notice says in part 
"The concerns relate to lack of transparency and misstatement of the hypotheses and predictions the reported meta-study was designed to test; lack of preregistration for measures and analyses supporting the titular claim (against statements asserting preregistration in the published article); selection of outcome measures and analyses with knowledge of the data; and incomplete reporting of data and analyses."

RETRACTED ARTICLE: High replicability of newly discovered social-behavioural findings is achievable

This article was retracted on 24 September 2024

Matters Arising to this article was published on 24 September 2024

This article has been updated

Abstract

Failures to replicate evidence of new discoveries have forced scientists to ask whether this unreliability is due to suboptimal implementation of methods or whether presumptively optimal methods are not, in fact, optimal. This paper reports an investigation by four coordinated laboratories of the prospective replicability of 16 novel experimental findings using rigour-enhancing practices: confirmatory tests, large sample sizes, preregistration and methodological transparency. In contrast to past systematic replication efforts that reported replication rates averaging 50%, replication attempts here produced the expected effects with significance testing (P < 0.05) in 86% of attempts, slightly exceeding the maximum expected replicability based on observed effect sizes and sample sizes. When one lab attempted to replicate an effect discovered by another lab, the effect size in the replications was 97% that in the original study. This high replication rate justifies confidence in rigour-enhancing methods to increase the replicability of new discoveries.

########


############

In general, I'm more optimistic about replications than preregistrations for identifying replicable results and testing hypotheses about them.  In this case, preregistration apparently revealed that what was written up as a replication study had begun as something else, and that the goal posts had been moved ex post, apparently in inappropriate ways.
######
Somewhat related are my posts on the Einstein Foundation Award for Promoting Quality in Research.

Saturday, October 5, 2024

The NAS proposes that bans on studying marijuana and its effects should be relaxed

 The National Academy of Sciences has just issued a new report on marijuana and public health.  Among their recommendations is that bans on research should be rescinded. (Because marijuana is currently a Schedule I drug in the Controlled Substances Act, it's hard to get permission to study it and its effects...)

Cannabis Policy Impacts Public Health and Health Equity (2024)

Monday, September 30, 2024

Golden Goose awards for woodpeckers, penguins, and artificial intelligence

 The Golden Goose awards are given each year to "recognize the tremendous human and economic benefits of federally funded research by highlighting examples of seemingly obscure studies that have led to major breakthroughs and resulted in significant societal impact."

This year they recognize three streams of work, that have led to the recovery of an endangered woodpecker species, to the more effective counting of penguins, and to the invention of neural nets on which the current artificial intelligence industry is based.

Here are those stories.

It’s a Family Affair: The Resurgence of the Red-Cockaded Woodpecker  AWARDEE: Jeff Walters

From Poop to Protection: Satellite Discoveries Help Save Antarctic Penguins and Advance Wildlife Monitoring  AWARDEES: Christian Che-Castaldo, Heather Joan Lynch, Mathew Schwaller

How We Think: Brain-Inspired Models of Human Cognition Contribute to the Foundations of Today’s Artificial Intelligence  AWARDEES: Geoffrey Hinton, James L. McClelland, David E. Rumelhart

Here's the first paragraph of the description of this third award (last but not least:)

"Decades before artificial intelligence emerged as the platform for innovation that it is today, David Rumelhart, James McClelland, and Geoffrey Hinton were exploring a new model to explain human cognition. Dissatisfied with the prevailing symbolic theory of cognition, David Rumelhart began to articulate the need for a new approach to modeling cognition in the mid-1970s, teaming up with McClelland with support from the National Science Foundation to create a model of human perception that employed a new set of foundational ideas. At around the same time, Don Norman, an early leader in the field of cognitive science, obtained funding from the Sloan Foundation to bring together an interdisciplinary group of junior scientists, including Hinton, with backgrounds in computer science, physics, and neuroscience. Rumelhart, McClelland, and Hinton led the development of the parallel distributed processing framework, also known as PDP, in the early-1980s, focusing on how networks of simple processing units, inspired by the properties of neurons in the brain, could give rise to human cognitive abilities. While many had dismissed the use of neural networks as a basis for building models of cognition in the 1960s and 1970s, the PDP group revived interest in the approach. Skeptics critiqued the new models too, and had only limited success in enabling effective artificially intelligent systems until the 2010s, when massive increases in the amount of available data and computer power enabled Hinton and others to achieve breakthroughs leading to an explosion of new technological advancements and applications."

############

Prior awards included market design in 2013 and 2014.


Monday, September 23, 2024

A 40 year old proof about top trading cycles is corrected (by two Berkeley grad students)

 Science (and math) can be self-correcting, sometimes slowly.  Here's an article that corrects the first proof that the top trading cycles algorithm is group strategy proof.  That's a true result, with multiple subsequent proofs.  But apparently the first proof presented wasn't the best one.  That's good to know.

One reason this may have taken a long time to spot is that the result is correct, and that there are subsequent proofs that connect the result to properties of other mechanisms.  

Will Sandholtz and Andrew Tai, the authors, did this work as Ph.D. students at UC Berkeley. (good for them!)

Group incentive compatibility in a market with indivisible goods: A comment  by Will Sandholtz and Andrew Tai

"Highlights

• Bird (1984), first to show top trading cycles is group strategy-proof, has errors.

•We present corrected results and proofs.

•We present a novel proof of strong group strategy-proofness without non-bossiness.

"Abstract: We note that the proofs of Bird (1984), the first to show group strategy-proofness of top trading cycles (TTC), require correction. We provide a counterexample to a critical claim and present corrected proofs in the spirit of the originals. We also present a novel proof of strong group strategy-proofness using the corrected results."

Saturday, March 30, 2024

Fraud in physics? Room temp superconductors, again

 It should come as no surprise that it's not only social sciences that can be roiled by accusations of research misconduct.

Here's a story in Nature about a scientist who had a paper retracted from Nature, and then had another accepted, and then also retracted, both about room temperature superconductors.  It's a long, detailed story, but it says something about both science and about peer review.

Superconductivity scandal: the inside story of deception in a rising star’s physics lab. Ranga Dias claimed to have discovered the first room-temperature superconductors, but the work was later retracted. An investigation by Nature’s news team reveals new details about what happened — and how institutions missed red flags.   By Dan Garisto

"A researcher at the University of Rochester in New York, Dias achieved widespread recognition for his claim to have discovered the first room-temperature superconductor, a material that conducts electricity without resistance at ambient temperatures. Dias published that finding in a landmark Nature paper1.

"Nearly two years later, that paper was retracted. But not long after, Dias announced an even bigger result, also published in Nature: another room-temperature superconductor2.

...

" Nature has since retracted his second paper2 and many other research groups have tried and failed to replicate Dias’s superconductivity results. ...The scandal “has damaged careers of young scientists...

...

"Three previous investigations ... by the University of Rochester did not find evidence of misconduct. But last summer, the university launched a fourth investigation,... That fourth investigation is now complete and, according to a university spokesperson, the external experts confirmed that there were “data reliability concerns” 

...

"Nature retracted the CSH paper on 26 September 2022, with a notice that states “issues undermine confidence in the published magnetic susceptibility data as a whole, and we are accordingly retracting the paper”.

...

"Felicitas Heβelmann, a specialist in retractions at the Humboldt University of Berlin, says misconduct is difficult to prove, so journals often avoid laying blame on authors in retractions. “A lot of retractions use very vague language,” she says.

...

"The lack of industry-wide standards for investigating misconduct leaves it unclear whether the responsibility to investigate lands more on journals or on institutions.

...

"After Nature published the LuH paper in March 2023, many scientists were critical of the journal’s decision, given the rumours of misconduct surrounding the retracted CSH paper.

...

"All four referees agreed that the findings, if true, were highly significant. But they emphasized caution in accepting the manuscript, because of the extraordinary nature of the claims. Referee 4 wrote that the journal should be careful with such extraordinary claims to avoid another “Schön affair”, referring to the extensive data fabrication by German physicist Jan Hendrik Schön, which has become a cautionary tale in physics and led to dozens of papers being retracted, seven of them in Nature. Referees 2 and 3 also expressed concern about the results because of the CSH paper, which at the time bore an editor’s note of concern but had not yet been retracted. 

...

"When asked why Nature considered Dias’s LuH paper after being warned of potential misconduct on the previous paper, Magdalena Skipper, Nature’s editor-in-chief, said: “Our editorial policy considers every submission in its own right.” The rationale, Skipper explains, is that decisions should be made on the basis of the scientific quality, not who the authors are."

Friday, January 19, 2024

Incentives and mis-incentives in science (Freakonomics part II)

 Freakonomics has a second post on fraud in science, and you can listen or read the transcript here:

Can Academic Fraud Be Stopped?

Two quotes stood out for me:

1. VAZIRE: Oh, I don’t mind being wrong. I think journals should publish things that turn out to be wrong. It would be a bad thing to approach journal editing by saying we’re only going to publish true things or things that we’re 100 percent sure are true. The important thing is that the things that are more likely to be wrong are presented in a more uncertain way. And sometimes we’ll make mistakes even there. Sometimes we’ll present things with certainty that we shouldn’t have. What I would like to be involved in and what I plan to do is to encourage more post-publication critique and correction, reward the whistleblowers who identify errors that are valid and that need to be acted upon, and create more incentives for people to do that, and do that well.

...

2. BAZERMAN: Undoubtedly, I was naive. You know, not only did I trust my colleagues on the signing-first paper, but I think I’ve trusted my colleagues for decades, and hopefully with a good basis for trusting them. I do want to highlight that there are so many benefits of trust. So, the world has done a lot better because we trust science. And the fact that there’s an occasional scientist who we shouldn’t trust should not keep us from gaining the benefit that science creates. And so one of the harms created by the fraudsters is that they give credibility to the science-deniers who are so often keeping us from making progress in society.


############

Earlier:

Sunday, January 14, 2024

Sunday, January 14, 2024

"Why Is There So Much Fraud in Academia?" Freakonomics interviews Max Bazerman and others

Below is the latest Freakonomics podcast (and transcript), on fraud in academia.  Those most in the headlines weren't available to be interviewed, but their coauthor (and my longtime HBS colleague) Max Bazerman gives his perspective.

Also interviewed are the Data Colada authors/data sleuths Leif Nelson Uri Simonsohn, and Joe Simmons (with some clues about the name of their blog), and Brian Nosek, who founded the prizewinning Center for Open Science (https://www.cos.io/ 

Here it is:

Why Is There So Much Fraud in Academia?  Some of the biggest names in behavioral science stand accused of faking their results. Freakonomics EPISODE 572.

######

And here are two paragraphs from Max's HBS web page (linked above), suggesting more to come:

"I have been connected to one of the most salient episodes of data fabrication in the history of social science – involving the signing first effect alluded to above. I am working on understanding all known social science frauds in this millennium. Social science also struggles with a broader problem, namely the fact that many studies fail to replicate due to faulty research practices that have become common in social science. Most replication failures can be traced back to the original researchers twisting their data to conform to their predictions, rather than from outright fraud. Trying to produce “significant” results, they may run a study multiple times, in a variety of ways, then selectively report the tests that worked and fail to report those that didn’t. The result is the publication of conclusions that do not hold up as accurate. Both problems – outright data fabrication and this reporting bias that shapes results – need to be tackled, so all of us in academia can publish results that are replicable and can help create value in society.

         "The last dozen years have witnessed multiple efforts to reform social science research to make it more credible, reproducible, and trusted. I am writing a book on reforming social science, which will provide an account of recent data fabrications, and highlight strategies to move forward to create more credible and impactful scientific research."

Wednesday, November 15, 2023

2023 EINSTEIN FOUNDATION AWARD FOR PROMOTING QUALITY IN RESEARCH

 Here's the press release from the Einstein Foundation in Berlin:

THE WINNERS OF THE 2023 EINSTEIN FOUNDATION AWARD FOR PROMOTING QUALITY IN RESEARCH

"THE EINSTEIN FOUNDATION BERLIN IS TO HONOR BELGIAN BIOINFORMATICIAN YVES MOREAU, THE BERKELEY INITIATIVE FOR TRANSPARENCY IN THE SOCIAL SCIENCES, AND THE RESPONSIBLE RESEARCH ASSESSMENT INITIATIVE WITH THIS YEAR’S EINSTEIN FOUNDATION AWARD FOR PROMOTING QUALITY IN RESEARCH 2023. 

The recipient of the Individual Award is Yves Moreau from the Katholieke Universiteit Leuven. Moreau ranks among the most ardent advocates for ethical standards in the utilization of human DNA data in the age of artificial intelligence and big data. He designs algorithms that protect personal privacy during the analysis of genetic data. This year’s Institutional Award recognizes the work of the Berkeley Initiative for Transparency in the Social Sciences (BITSS), which advocates for rigor, transparency, and reproducibility in social scientific research. The Institute achieves this through establishing open science practices, developing appropriate infrastructure, and conducting meta-research. The 2023 Early Career Award goes to the Responsible Research Assessment Initiative headed by Anne Gärtner (Dresden University of Technology). The project aims to identify, test, and establish novel criteria for the assessment of researchers and their output. Moving away from quantity of output and other unsuitable metrics, it will foreground quality of research by taking into account factors such as transparency, robustness, innovation, and cooperation. 

...

"Jury member Michel Cosnard, computer scientist at the Université Nice-Sophia-Antipolis, believes that Yves Moreau is highly deserving of the award, which recognizes his unwavering dedication on both professional and ethical fronts. “Moreau links deep research in DNA analysis and artificial intelligence with ethics, integrity, and human rights. His work and achievements serve as a cornerstone to help us confront the difficult social questions that arise from rapid technological developments.” 

"Fellow jury member and Stanford University economist Alvin Roth firmly endorses the chosen winner of the Institutional Award: “The Berkeley Initiative for Transparency in the Social Sciences plays an active, creative role in the ‘credibility revolution’ in science by promoting careful experimentation, and supporting efforts to make replication and verification commonplace.” 

###########

The two previous awards

Monday, December 5, 2022

Thursday, May 18, 2023

Pushback against high prices in academic publishing: open access fees and e-textbooks to libraries

 The Guardian has the story:

‘Too greedy’: mass walkout at global science journal over ‘unethical’ fees. Entire board resigns over actions of academic publisher whose profit margins outstrip even Google and Amazon  by Anna Fazackerley

"The entire academic board of the journal Neuroimage, including professors from Oxford University, King’s College London and Cardiff University resigned after Elsevier refused to reduce publication charges.

Academics around the world have applauded what many hope is the start of a rebellion against the huge profit margins in academic publishing, which outstrip those made by Apple, Google and Amazon.

Neuroimage, the leading publication globally for brain-imaging research, is one of many journals that are now “open access” rather than sitting behind a subscription paywall. But its charges to authors reflect its prestige, and academics now pay over £2,700 for a research paper to be published. The former editors say this is “unethical” and bears no relation to the costs involved.

...

"Elsevier, a Dutch company that claims to publish 18% of the world’s scientific papers, reported a 10% increase in its revenue to £2.9bn last year. But it’s the profit margins, nearing 40%, according to its 2019 accounts, which anger academics most. The big scientific publishers keep costs low because academics write up their research – typically funded by charities and the public purse – for free. They “peer review” each other’s work to verify it is worth publishing for free, and academic editors collate it for free or for a small stipend. Academics are then often charged thousands of pounds to have their work published in open-access journals, or universities will pay very high subscription charges.

...

"Meanwhile, university libraries are angry about the cost of the online textbooks they say students now overwhelmingly want to read – often many times more expensive than their paper equivalent. Professor Chris Pressler, director of Manchester University Library, said: “We are facing a sustained onslaught of exploitative price models in both teaching and research.”

"According to a spreadsheet of costs quoted to university librarians, Manchester University gave a recent example of being quoted £75 for a popular plant biology textbook in print, but £975 for a three-user ebook licence. Meanwhile Learning to Read Mathematics in the Secondary School, a textbook for trainee teachers published by Routledge, was £35.99 in print and £560 for a single user ebook.

"A spokesperson for Taylor and Francis, which owns Routledge, said: “We strive to ensure that book prices are both affordable and a fair representation of their value.” He said a print book could be checked out for weeks at a time whereas ebooks could be checked in and out rapidly and had a much wider distribution."

Sunday, May 14, 2023

Morality in Economics, as viewed from Sociology (Georg Kanitsar in European J. of Sociology)

Georg Kanitsar, a young sociologist, undertakes the task of looking at how economists think about morality (with a focus on experimental and behavioral econ, and market design). His view of how economists think may shed some light (for economists) on how sociologists think. (I quote below from near the beginning and near the end of his paper.)

Kanitsar, G. (2023). Putting Morals into Economics: From Value Neutrality to the Moral Economy and the Economization of Morality. European Journal of Sociology / Archives Européennes De Sociologie, 1-30. doi:10.1017/

"Abstract: The economic discipline plays a performative role in constructing the moral order of market society. Yet, little attention has been paid to what economists explicitly regard as moral or how they conceive of morality. This article reflects a recent attempt to put morals into economics, that is, to introduce morality as a research topic in behavioural and experimental economics. It maps three research programs that theorize the moral economy. The programs emphasize the moral foundations of market society, the moral limits of market expansion, and the moral consequences of market trading and, thus, appear irreconcilable with classifications of economists as market enthusiasts or moral agnostics. At the same time, however, the literature centres on an “economized” form of morality that is corrective to market inefficiencies, attributed to the responsibility of the individual, and expressed in rational terms. In doing so, this literature contributes to redefining moral problems in economic terms."

"I consider efforts to incorporate morality into an economic framework advanced by two influential branches of the discipline—behavioural economics and market experiments. To gain an overview of relevant research in these branches, I assemble a database of 39 recent articles and identify 20 key articles among them.

...

I explore the “economization of morality” by elucidating the moral arguments and the moral background of two authoritative programs in present-day economics: behavioural economics/experiments and market design/experiments. While many renowned economists have produced notable work on morality, these two research programs currently exert a unique influence on the economic discipline and are highly industrious in exporting its findings to policy making.

...

"Discourse in the economic mainstream was long dominated by market enthusiasts and moral agnostics, but the recent surge of behavioural and market experiments has again drawn attention to morality as a research topic in economics. At the argumentative level, the reviewed literature reveals a genuine break with market fundamentalism in the narrow sense. I have identified three strands that shed light on the moral economy and emphasize the moral foundations, limitations, and consequences of markets. Thus, economics has not been deaf to appeals to “put morals into markets” [Amable Reference Amable2011]. At the background level, however, the integration of morality is steered by the discipline’s theoretical and methodological underpinnings. In consequence, a very specific understanding of morality lies at the heart of these research efforts; a form of morality that is functional to market efficiency and attributed to utility-maximizing individuals.

...

"Behavioural economics thus strikes out in the opposite direction as scholarship in economic sociology. On the surface, both disciplines take as a starting point a view of market society as divided in arm’s-length transactions and social ties, and both disciplines have rediscovered morality as their subject matter. Yet, behavioural economics addresses the social sphere with tools that were tailor-made for the neoclassical analysis of markets. The field maintains the analytical primacy on efficiency and rationality, which it inherited from its parent discipline. In experiments, social exchanges are represented as contractual, anonymous, and temporary encounters, and money is regarded as a neutral tool used to express valuations. Conversely, economic sociology views markets as diverse “arenas of social interaction” [Beckert Reference Beckert2009: 245]. Market transactions are considered as far from universal, arelational, and disembedded [Aspers Reference Aspers, Beckert and Zafirovski2005], and the cultural meanings of money rarely reduce it to a qualityless, neutral, and homogenous medium of exchange [Zelizer Reference Zelizer1989]. Thus, the “moral economy” of behavioural economics is situated next to the “amoral economy” of neoclassical economics [Bowles Reference Bowles2016], echoing the traditional opposition between separate spheres of the economic and the moral [Thompson Reference Thompson E1971]. By contrast, economic sociology is increasingly devoted to identifying the multiple moralities underlying economic processes [Beckert Reference Beckert2012; Zelizer Rotman Reference Zelizer Rotman2017], convinced that “all economies are moral economies” [Fourcade Reference Fourcade2017: 665]."

Saturday, February 11, 2023

Monday, December 5, 2022

The 2022 Einstein Foundation Awards

 Here are the Recipients of the 2022 Einstein Foundation Awards

Einstein Foundation Individual Award Winner 2022:

Gordon Guyatt, for his seminal contributions to evidence based medicine (including the name)

"Einstein Foundation Award winner Gordon Guyatt improved the quality of clinical research and helped bring evidence to medicine. His next mission is bringing patients into the discussion"


Einstein Foundation Institutional Award 2022:

The Psychological Science Accelerator

"The Psychological Science Accelerator, winner of the 2022 Institutional Award, has made its name by facilitating large, crowdsourced international studies in all aspects of the discipline and by democratizing and diversifying big team psychological science."


Einstein Foundation Early Career Award 2022:

Ape Research Index (the name almost speaks for itself)


Here's the description of the Award(s): The Einstein Foundation Award for Promoting Quality in Research

"The Einstein Foundation Award for Promoting Quality in Research aims to provide recognition and publicity for outstanding efforts that enhance the rigor, reliability, robustness, and transparency of research in the natural sciences, the social sciences, and the humanities, and stimulate awareness and activities fostering research quality among scientists, institutions, funders, and politicians. To acknowledge the outstanding role early career researchers (ECRs) have in promoting research quality, ECRs will be invited to propose projects that foster research quality and value. Projects will be competitively selected for funding and internationally showcased."

Here are the members of this year's award jury. (I am one of the 15 jury members).


Here's my post on last year's prizes:

Thursday, November 25, 2021


Friday, October 14, 2022

Scientific honors and gender gaps

 Changing from an old equilibrium to a new one can involve some actions that may not persist once a new equilibrium is reached.  Here's a paper on the awarding of scientific honors.

Gender Gaps at the Academies by David Card, Stefano DellaVigna, Patricia Funk & Nagore Iriberri

NBER WORKING PAPER 30510  DOI 10.3386/w30510  September 2022

"Historically, a large majority of the newly elected members of the National Academy of Science (NAS) and the American Academy of Arts and Science (AAAS) were men. Within the past two decades, however, that situation has changed, and in the last 3 years women made up about 40 percent of the new members in both academies. We build lists of active scholars from publications in the top journals in three fields – Psychology, Mathematics and Economics – and develop a series of models to compare changes in the probability of selection of women as members of the NAS and AAAS from the 1960s to today, controlling for publications and citations. In the early years of our sample, women were less likely to be selected as members than men with similar records. By the 1990s, the selection process at both academies was approximately gender-neutral, conditional on publications and citations. In the past 20 years, however, a positive preference for female members has emerged and strengthened in all three fields. Currently, women are 3-15 times more likely to be selected as members of the AAAS and NAS than men with similar publication and citation records."

************

That paper is a followup to their previous paper on Econometric Society Fellows, which has just come out in the current issue of Econometrica:

ECONOMETRICA: SEP 2022, VOLUME 90, ISSUE 5

Gender Differences in Peer Recognition by Economists

https://doi.org/10.3982/ECTA18027
p. 1937-1971

David Card, Stefano DellaVigna, Patricia Funk, Nagore Iriberri

We study the selection of Fellows of the Econometric Society, using a new data set of publications and citations for over 40,000 actively publishing economists since the early 1900s. Conditional on achievement, we document a large negative gap in the probability that women were selected as Fellows in the 1933–1979 period. This gap became positive (though not statistically significant) from 1980 to 2010, and in the past decade has become large and highly significant, with over a 100% increase in the probability of selection for female authors relative to males with similar publications and citations. The positive boost affects highly qualified female candidates (in the top 10% of authors) with no effect for the bottom 90%. Using nomination data for the past 30 years, we find a key proximate role for the Society's Nominating Committee in this shift. Since 2012, the Committee has had an explicit mandate to nominate highly qualified women, and its nominees enjoy above‐average election success (controlling for achievement). Looking beyond gender, we document similar shifts in the premium for geographic diversity: in the mid‐2000s, both the Fellows and the Nominating Committee became significantly more likely to nominate and elect candidates from outside the United States. Finally, we examine gender gaps in several other major awards for U.S. economists. We show that the gaps in the probability of selection of new fellows of the American Academy of Arts and Sciences and the National Academy of Sciences closely parallel those of the Econometric Society, with historically negative penalties for women turning to positive premiums in recent years.

**********

Update: here's the published version of the NBER paper, in PNAS, JANUARY 24, 2023, VOL. 120, NO. 4:

Gender gaps at the academies

David Card https://orcid.org/0000-0001-6914-6698 card@econ.berkeley.edu, Stefano DellaVigna, Patricia Funk, and Nagore Iriberri

This contribution is part of the special series of Inaugural Articles by members of the National Academy of Sciences elected in 2021. January 19, 2023 120 (4) e2212421120

https://doi.org/10.1073/pnas.2212421120

Saturday, August 6, 2022

Market design in an historical perspective, by Basshuysen in J. Economic Methodology

 I pay some attention to how market design is discussed in the history/sociology of science and methodology/history of thought literatures (even though I know how Feynman thought they are related to ornithology).

 Here's a pretty sympathetic, modern view.

Philippe van Basshuysen (2022) Markets, market algorithms, and algorithmic bias, Journal of Economic Methodology, DOI: 10.1080/1350178X.2022.2100919

ABSTRACT: "Where economists previously viewed the market as arising from a ‘spontaneous order’, antithetical to design, they now design markets to achieve specific purposes. This paper reconstructs how this change in what markets are and can do came about and considers some consequences. Two decisive developments in economic theory are identified: first, Hurwicz’s view of institutions as mechanisms, which should be designed to align incentives with social goals; and second, the notion of marketplaces – consisting of infrastructure and algorithms – which should be designed to exhibit stable properties. These developments have empowered economists to create marketplaces for specific purposes, by designing appropriate algorithms. I argue that this power to create marketplaces requires a shift in ethical reasoning, from whether markets should reach into certain spheres of life, to how market algorithms should be designed. I exemplify this shift, focusing on bias, and arguing that transparency should become a goal of market design"

I have to admit, I found this section heading charming (and not as funny as everyone else should:)

"2. What are markets, what can they do? From Hayek to Hurwicz to Roth"


Sunday, April 10, 2022

Transplant science, transplant politics , and public perception--the case of living-donor livers

 Liver transplants from living donors are on the increase in the U.S., so liver transplants are coming to more closely resemble kidney transplants.  Apparently that comes with some of the politics that I've witnessed involving kidney transplants. At least that's what the article below, by a very large and distinguished set of American authors, seems to be dealing with, in this critique of a paper published in a European journal. (It's unusual to see a question about scientific integrity highlighted in the title of an academic paper...). Apparently the title of the paper being criticized received news coverage unfavorable to transplantation, and not justified by the data being reported.

Emamaullee, J., Heimbach, J.K., Olthoff, K.M., Pomfret, E.A., Roberts, J.P., Selzner, N. and NALLDIG Consortium, 2022. Assessment of long‐term outcomes post living liver donation highlights the importance of scientific integrity when presenting transplant registry data. American Journal of Transplantation, online early,  First published: 30 March 2022 https://doi.org/10.1111/ajt.17045

"Abstract: Living donor liver transplantation has expanded in recent years, particularly in North America. As experience with this procedure has matured over the last 25 years, centers are increasingly faced with potential living donors who are more medically complex. As donors move through the evaluation process, completing the informed consent process continues to be challenged by a paucity of granular data demonstrating long-term outcomes and overall safety specifically in the otherwise ‘healthy’ living liver donor population. Two recently published studies examined long-term outcomes post-living liver donation using Korean registry data and reported similar results, with excellent overall survival when compared to appropriately matched controls. However, the authors of these studies were presented differently, with one reporting an alarmist view based on one aspect of a suboptimal analysis approach using an inappropriate comparator group. Herein, the North American Living Liver Donor Innovation Group (NALLDIG) consortium discusses these two studies and their potential impact on living liver donation in North America, ultimately highlighting the importance of scientific integrity in data presentation and dissemination when using transplant registry data."


"Recently, in the Journal of Hepatology, Choi and colleagues published a noteworthy study of long-term outcomes of Korean living liver donors, titled “Outcomes of living liver donors are worse than those of matched healthy controls.”

...

"Although careful examination of long-term outcomes for living liver donors is essential, and the South Korean population is an excellent population in which to conduct these analyses given the relative frequency of living donor liver transplantation, the current study suffers from a serious design flaw which makes interpretation of the data very challenging, as highlighted in detail in the letter to the editor from the Toronto group.4 Indeed, the Control Group I “heathy population” is actually healthier than the living liver donors given that “the proportion of individuals with Charlson Comorbidity Index ≥1, diabetes, hypertension, or depression in the living liver donor group was higher than that in the matched healthy group (Control Group I).” They then use this heavily weighted (10:1 match) heathier group to demonstrate, unsurprisingly, a subtle but statistically worse outcome in the living liver donor group, hence the title of the paper. 

...

"Unfortunately, both the manuscript title, “Outcomes of living liver donors are worse than those of matched healthy controls” and the lay summary, which is intended for the public at large are highly misleading, given this lack of a matched control group. This leads to confusion and uncertainty not only for those within the medical community but also for prior donors as well as for future potential living donors and their recipients. One may wonder whether these choices were made to drive the sensationalism of the media on this topic particularly in some European countries where practice of LDLT is no longer pursued for a variety of reasons including overall skepticism that the procedure is truly safe for living donors.

...

"Journal editorial boards bear the responsibility of arranging high-quality, expert peer review. The dramatic increase in the volume of manuscripts submissions during COVID-19 pandemic has strained many editorial board members and reviewers, likely resulting in publication of studies of variable rigor.16 In parallel, careful assessment of statistical techniques and interpretation of analyses has become an integral part of the review process. Many journals now employ full-time statistical editors, which has enabled identification of serious flaws in experimental design including insufficient study power, missing data, or inappropriate use of statistical tests or models.17 Editors must also critically assess the power of a provocative manuscript title, as initially it may grab the reader's attention and can bias the reader's interpretation and impression. Journals also must be willing to publish studies even when results are disappointing and/or controversial. The transplant community should continue to carefully assess and respond to any work that may dampen enthusiasm for living donation or living donor liver transplantation."