Search engines as modern oracles, by Andrei Z. Broder & Preston McAfee

 Among the costs of visiting Mt. Parnassus to consult the Oracle at Delphi was that it was difficult to interpret the answers received, particularly if the questions were posed too casually.  Modern search engines are a bit like ancient oracles that way, and the resemblance may become greater as generative AI chatbots based on Large Language Models (LLMs) become part of search.

Here is some advice about how to think about all that.

Delphic Costs and Benefits in Web Search: A utilitarian and historical analysis. by Andrei Z. Broder & Preston McAfee, Google Research, August 16, 2023

Abstract: "We present a new framework to conceptualize and operationalize the total user experience of search, by studying the entirety of a search journey from an utilitarian point of view.

"Web search engines are widely perceived as “free”. But search requires time and effort: in reality there are many intermingled nonmonetary costs (e.g. time costs, cognitive costs, interactivity costs) and the benefits may be marred by various impairments, such as misunderstanding and misinformation. This characterization of costs and benefits appears to be inherent to the human search for information within the pursuit of some larger task: most of the costs and impairments can be identified in interactions with any web search engine, interactions with public libraries, and even in interactions with ancient oracles. To emphasize this innate connection, we call these costs and benefits Delphic, in contrast to explicitly financial costs and benefits.

"Our main thesis is that users’ satisfaction with a search engine mostly depends on their experience of Delphic cost and benefits, in other words on their utility. The consumer utility is correlated with classic measures of search engine quality, such as ranking, precision, recall, etc., but is not completely determined by them. To argue our thesis, we catalog the Delphic costs and benefits and show how the development of search engines over the last quarter century, from classic Information Retrieval roots to the integration of Large Language Models, was driven to a great extent by the quest of decreasing Delphic costs and increasing Delphic benefits.

"We hope that the Delphic costs framework will engender new ideas and new research for evaluating and improving the web experience for everyone."

And here's the final paragraph:

"We hope that this paper will engender new ideas for Delphic costs assessments, the measurement of Delphic costs, and means of reducing these costs. We would like to see the evaluation of web search engines move away from assessing the quality of ranking in isolation of the users’ overall search experience and personal context towards a holistic evaluation of user utility from using search engines. Moreover, this “utilitarian analysis” approach, rather than pure relevance analysis, could and should be applied to situations that do not involve explicit search, such as content feeds and recommender systems."

Pitfalls of digital scholarship: Machiavelli and Matching

 One of the alluring features of the digitization of texts is that they can be searched, their citations can be examined and cross-referenced, and facts about texts, and the literatures that they comprise, can be detected.  But of course,  digital searches can also lead you astray.

Something like that may have happened in this study of business ethics. (Relax, this isn't a blog post about questionable ethics in science.)  

Maity, M., Roy, N., Majumder, D. et al. Revisiting the Received Image of Machiavelli in Business Ethics Through a Close Reading of The Prince and Discourses. J Bus Ethics (2023). https://doi.org/10.1007/s10551-023-05481-2

The authors of the above paper searched in journals related to business and economics, for papers  about Niccolò Machiavelli, the 16th century author of The Prince, whose name has entered into the language to describe the kind of advice he gave: Machiavellian.

Looking at the most highly cited papers, and their network of co-citations (i.e. citations of each other) they find three clusters in the Machiavelli literature. They note that two of the clusters include many citations from one to the other, but that the third cluster (in green) is not connected to the other two.  The third cluster they label "matching problems in markets." (In fairness, the authors of the paper note this separation, and concentrate their analysis on the first two clusters.)

Here are the papers in the clusters. The papers in cluster 3 will be familiar to many readers of this blog.

Here in larger font is cluster 3, of papers on "Matching problems in markets": Abdulkadiroǧlu et al. (2003), Abdulkadiroǧlu and Sönmez (2003), Dubins and Freedman, (1981), Gale and Shapley (1962), Gale and Sotomayor (1985a), Gale and Sotomayor (1985b), Kojima and Pathak (2009), Roth (1982, 1984a, 1984b, 1985, 2002), Roth and Sotomayor (1990), Roth and Peranson (1999).

This cluster indeed contains well cited papers that cite one another. Yet I'm pretty sure that none of them cite Machiavelli, nor would most readers think that they connect to The Prince.

This latter cluster was almost surely included because of the titles of two of the included papers, neither of which in fact cites Machiavelli. (His name made it into the titles in a sort of jokey way, having to do with the fact that players in matching games may sometimes profit from behaving unstraightforwardly.) They are:

Dubins, Lester E., and David A. Freedman. "Machiavelli and the Gale-Shapley algorithm." The American Mathematical Monthly 88, no. 7 (1981): 485-494.

and

Gale, David, and Marilda Sotomayor. "Ms. Machiavelli and the stable matching problem." The American Mathematical Monthly 92, no. 4 (1985): 261-268.

But Machiavelli might be proud to be included in an economic literature on incentives.

CHOICE SCREEN AUCTIONS by Michael Ostrovsky

 Mike Ostrovsky points out that small design decisions can have big consequences, and considers how European regulations have caused search engines to be allocated on Android phones.

CHOICE SCREEN AUCTIONS by Michael Ostrovsky, NBER Working Paper http://www.nber.org/papers/  (a less gated version is here)

"ABSTRACT: Choice screen auctions have been recently deployed in 31 European countries, allowing consumers to choose their preferred search engine on Google's Android platform instead of being automatically defaulted to Google's own search engine. I show that a seemingly minor detail in the design of these auctions—whether they are conducted on a “per appearance” or a “per install” basis—plays a major role in the mix and characteristics of auction winners, and, consequently, in their expected overall market share. I also show that “per install” auctions distort the incentives of alternative search engines toward extracting as much revenue as possible from each user who installs them, at the expense of lowering the expected number of such users. The distortion becomes worse as the auction gets more competitive and the number of bidders increases. Empirical evidence from Android choice screen auctions conducted in 2020 is consistent with my theoretical results."

The auction rules: "In each country auction, search providers will state the price that they are willing to pay each time a user selects them from the choice screen in the given country. The three highest bidders will appear in the choice screen for that country. The provider that is selected by the user will pay the amount of the fourth-highest bid."

...

"In this paper, I show that a seemingly minor detail of the implementation of choice screen auctions plays a major role in their outcomes—and thus in the overall effectiveness of the antitrust remedy. Specifically, while the answer in the Q&A section of the document states that an auction “allows search providers to decide what value they place on appearing in the choice screen and to bid accordingly,” the auction, as implemented, charges these providers not for appearing in the choice screen but for being chosen by a user. 

"While the difference may seem to be just a matter of language, it is not. To see the intuition for the difference, consider a version of the auction with just one available spot and two bidders. Bidder A gets revenue $10 from each user who installs its search engine, and if it is shown as an option in the choice screen, then the probability that a user will choose it is 10%. Bidder B gets revenue $20 from each user who installs its search engine, but the probability that a user will choose it (if it is shown as an option in the choice screen) is only 1%. The value that bidder A has for appearing on the screen is therefore $1, and the value that bidder B has for appearing on the screen is $0.20. Thus, if the auction is conducted on the “per appearance” basis, then bidder A will win, will pay $0.20 per appearance, and will have its search engine chosen by users 10% of the time, while the dominant platform’s own search engine will be chosen 90% of the time. If, instead, the auction is conducted as implemented, with bidding and payment on the “per install” basis, then bidder B will win and will pay $10 every time its search engine is chosen (corresponding to $0.10 per appearance). The winner’s search engine will be chosen only 1% of the time, and the dominant platform’s one will be chosen the remaining 99% of the time. Thus, relative to the per appearance auction, the per install auction results in a lower likelihood that an alternative search engine will be chosen by the user (making it correspondingly more attractive to the dominant platform) and gives advantage to search engines that generate higher revenue per user vs. those that are more popular but generate less revenue on a per-user basis. I