Eliciting beliefs is a difficult task: Danz, Vesterlund and Wilson in the AER

 Experimental economists have long been faced with the difficulty of eliciting coherent beliefs from participants in experiments, even when they seem to act in a way consistent with having coherent beliefs.  This is consistent with the psychological view that beliefs in the full Bayesian sense may not closely correspond to people's internal psychological representation of the world. So techniques for belief elicitation that would work well for idealized utility maximizers may not be ideal for human subjects.

Here's a thoughtful effort to come to grips with that.

Belief Elicitation and Behavioral Incentive Compatibility  by David Danz, Lise Vesterlund, Alistair J. Wilson AMERICAN ECONOMIC REVIEW  VOL. 112, NO. 9, SEPTEMBER 2022  (pp. 2851-83)

Abstract: "Subjective beliefs are crucial for economic inference, yet behavior can challenge the elicitation. We propose that belief elicitation should be incentive compatible not only theoretically but also in a de facto behavioral sense. To demonstrate, we show that the binarized scoring rule, a state-of-the-art elicitation, violates two weak conditions for behavioral incentive compatibility: (i) within the elicitation, information on the incentives increases deviations from truthful reporting; and (ii) in a pure choice over the set of incentives, most deviate from the theorized maximizer. Moreover, we document that deviations are systematic and center-biased, and that the elicited beliefs substantially distort inference."

"We argue that to secure truthful revelation, elicitation mechanisms need to not only be incentive compatible in a purely theoretical sense, but also in a behavioral one. We propose for assessment two weak conditions for behavioral incentive compatibility, that information on deployed incentives increases truthful revelation; and that most participants, when given a choice over the pure incentives, select the outcome thought to be uniquely maximizing under the mechanism (i.e., a requirement of behavioral incentive compatibility for a representative agent).

"To demonstrate we explore a state-of-the-art belief elicitation, the binarized scoring rule (BSR) (Hossain and Okui 2013). The BSR is seen as a particularly promising alternative to elicitations requiring risk neutrality because its incentive compatibility expands to arbitrary EU preferences—in fact, to any decision-maker who maximizes the overall chance of winning a prize. Building on the insights of Roth and Malouf (1979), this is achieved by linking reported beliefs to a pair of state-contingent lotteries, where for each distinct belief, the mechanism provides a lottery pair with a stochastically dominant reduction. That is, decision-makers who maximize their chance of winning are given strict incentivizes under the BSR to reveal their true belief.

"In addition to being incentive compatible for a wider set of preferences, initial empirical evidence shows that the BSR outperforms its narrower forerunner, the quadratic scoring rule (Hossain and Okui 2013; Harrison and Phillips 2014). Weakened theoretical assumptions and evidence for superior relative performance has quickly rendered the BSR the preferred elicitation.3  However, limited evidence exists on whether subjects behave in a truth-telling manner, and the conservative reporting patterns that identified failures in quadratic-scoring elicitations have also been detected in BSR elicitations. For example, in Babcock et al. (2017), despite the qualitative comparative statics for beliefs mirroring behavior, the elicited reports appeared overly conservative."

...[and from the conclusion]

"In pursuing improved elicitations, we need to be cognizant that we are designing mechanisms for behavioral agents. In this respect, our findings and proposed tests for behavioral incentive compatibility relate to Li’s (2017) concepts of obvious dominance and obvious strategy proofness. Both our work and Li (2017) stress the importance of considering cognitive limitations (in addition to a broader set of preferences) when designing incentive compatible mechanisms. However, while Li (2017) provides a theoretical criterion of a mechanism’s incentive compatibility for a class of cognitively limited agents, our work stresses the importance of, and provides means to, testing whether a theoretically incentive compatible mechanism is behaviorally incentive compatible in an empirical sense. As in the BSR, relatively weak-seeming theoretical assumptions permit the design of fully separating mechanisms, to measure beliefs at arbitrary precision. But such precision may well be costly—where we need to empirically test that the assumptions put in place hold, and that behavioral agents actually perceive truthful revelation as beneficial.

"Our study has proposed weak conditions for behaviorally incentive compatible elicitations and provided diagnostic tools for checking them. The hope is that new elicitations will be assessed against and succeed in passing these standards. Given the challenges associated with this task though, we caution that it may be time to question whether it is reasonable to assume that participants in our studies hold exact probabilistic beliefs, let alone our ability to use monetary incentives to elicit such beliefs at arbitrary precision. Instead of taking our results as a call for the development of mechanisms that are incentive compatible for an ever-more-general class of decision-maker, we might instead ask whether the necessary economic inferences could be drawn with less-precise measurements, where the incentives for truthful reporting can be simpler and starker.63 For example, in discrete settings it may be sufficient to elicit the event the participants deem most likely and incentivize the elicitation by offering compensation only in the event that the report is correct.64 In continuous settings, the same can be achieved by paying participants if the true population outcome falls within some bounds around their guess.65 Alternatively, it may be sufficient to determine whether a belief lies within a certain fixed interval. This allows for deviations between the potential intervals to come at a higher perceived cost and may still provide the information necessary for inference.66 For example, suppose that in understanding individual behavior we wish to elicit the belief that an opponent will select action A or B, and that the individual’s predicted behavior theoretically depends on the belief on A exceeding a 30 percent cutoff. Rather than eliciting the precise belief that action A is chosen, it may secure more reliable and truthful reporting to instead focus the elicitation on whether or not the belief on A exceeds the theoretical cutoff. If elicited beliefs are collected primarily as controls or for auxiliary tests of a behavioral mechanic, inference may be improved with starker incentives over coarser elicitations.

"While there are many paths to improve belief elicitation, we propose two simple assessments: that information on the incentives increases truthful reporting, andt hat most participants when given a choice over the pure set of incentives select the theorized maximizer. In demonstrating the very substantial inferential consequences from using biased elicitations, our results serve as a call for elicitations to be incentive compatible both theoretically and behaviorally, but also as a strong caution against elicitations that rely on incentives that decrease truthful reporting."

**********

One quick thought I have is that when binary lottery games were introduced in Roth, A. E., & Malouf, M. W. (1979), it was to allow the predictions of utility maximizing theories to be precisely specified, rather than to control the behavior of the experimental subjects.

*********

Not so long ago I posted about another paper in the AER that deals with simple and robust ways of eliciting beliefs about others' behavior.

Tuesday, March 1, 2022

Targeting High Ability Entrepreneurs Using Community Information: Mechanism Design in the Field By Reshmaan Hussam, Natalia Rigol, and Benjamin N. Roth

"Mechanisms designed to elicit truthful reporting in the laboratory sometimes are cumbersome to administer and difficult to explain.  Here's a paper that finds that simple attempts to incentivize truthful reporting (including allowing other participants to hear each report, as well as small payments for reports that conform to community consensus) can help eliminate incentives to boost family and friends, when reports concern who could make most effective use of a cash grant."