Search engines as modern oracles, by Andrei Z. Broder & Preston McAfee

 Among the costs of visiting Mt. Parnassus to consult the Oracle at Delphi was that it was difficult to interpret the answers received, particularly if the questions were posed too casually.  Modern search engines are a bit like ancient oracles that way, and the resemblance may become greater as generative AI chatbots based on Large Language Models (LLMs) become part of search.

Here is some advice about how to think about all that.

Delphic Costs and Benefits in Web Search: A utilitarian and historical analysis. by Andrei Z. Broder & Preston McAfee, Google Research, August 16, 2023

Abstract: "We present a new framework to conceptualize and operationalize the total user experience of search, by studying the entirety of a search journey from an utilitarian point of view.

"Web search engines are widely perceived as “free”. But search requires time and effort: in reality there are many intermingled nonmonetary costs (e.g. time costs, cognitive costs, interactivity costs) and the benefits may be marred by various impairments, such as misunderstanding and misinformation. This characterization of costs and benefits appears to be inherent to the human search for information within the pursuit of some larger task: most of the costs and impairments can be identified in interactions with any web search engine, interactions with public libraries, and even in interactions with ancient oracles. To emphasize this innate connection, we call these costs and benefits Delphic, in contrast to explicitly financial costs and benefits.

"Our main thesis is that users’ satisfaction with a search engine mostly depends on their experience of Delphic cost and benefits, in other words on their utility. The consumer utility is correlated with classic measures of search engine quality, such as ranking, precision, recall, etc., but is not completely determined by them. To argue our thesis, we catalog the Delphic costs and benefits and show how the development of search engines over the last quarter century, from classic Information Retrieval roots to the integration of Large Language Models, was driven to a great extent by the quest of decreasing Delphic costs and increasing Delphic benefits.

"We hope that the Delphic costs framework will engender new ideas and new research for evaluating and improving the web experience for everyone."

And here's the final paragraph:

"We hope that this paper will engender new ideas for Delphic costs assessments, the measurement of Delphic costs, and means of reducing these costs. We would like to see the evaluation of web search engines move away from assessing the quality of ranking in isolation of the users’ overall search experience and personal context towards a holistic evaluation of user utility from using search engines. Moreover, this “utilitarian analysis” approach, rather than pure relevance analysis, could and should be applied to situations that do not involve explicit search, such as content feeds and recommender systems."