As the rankings of universities conducted by the magazine US News and World Reports have become more influential, there are a growing number of reports of the ways, fair and not so fair, that universities respond to what USNWR tries to measure.
Clemson University has been in the news in connection with their stated efforts to rise higher in the US News and World Report rankings of colleges.
They and their critics agree that they want to do this; the question is are they doing it in the right way for the right reasons.
Here's a critic who says no:Researcher Offers Unusually Candid Description of University's Effort to Rise in Rankings:
"Clemson University is run in an almost single-minded direction, with nearly all policies driven by how they will help the land-grant institution rise in U.S. News & World Report’s rankings, according to a university official whose candid comments stirred debate among conference-goers here on Tuesday."
and the reply:
Clemson Assails Allegations That It Manipulates 'U.S. News' Rankings
"Clemson University, stung by charges by one of its own researchers that it willfully manipulates the U.S. News & World Report rankings, fired back on Wednesday, saying the accusations are “outrageous” examples of “urban legends” that have surrounded the university’s campaign to reach the top 20 of public research universities.“The accusation that Clemson, its staff, and administrators have engaged in unethical conduct to achieve a higher ranking is untrue and unfairly disparages the sincere, unwavering, and effective efforts of faculty and staff to improve academic quality over the past 10 years,” reads a statement issued by the university’s chief spokeswoman, Catherine T. Sams. “While we have publicly stated our goal of a top-20 ranking, we have repeatedly stressed that we use the criteria as indicators of quality improvement and view a ranking as the byproduct, not the objective.” "
Here's a summary: Clemson Explains Its Approach to U.S. News Rankings
And here's a story about alleged simple mis-counting at USC's School of Engineering: More Rankings Rigging , and a summary reflecting the relation between what is measured and what is reported: Gaming the Rankings. Here's an illuminating paragraph:
"Any performance measure is ripe to be gamed. The percentage of alumni giving is a measure worth 5 percent of a ranking in U.S. News. A few years ago, Albion College made its own stir in the higher education rankings world when it increased its percentage of alumni making donations with the stroke of a pen. As The Wall Street Journal reported, the college recorded a $30 donation from a graduating senior as a $6 alumnus gift for the next five years. Clemson, in its systematic approach to raising its rank — “no indicator, no method, no process off limits to create improvement,” as Watt stated — solicited alumni donations in such a way as to increase their giving rate: Alumni were encouraged to give as little as $5 annually."
Note incidentally that there are different ways to try to rise in the rankings, and some may be strictly gaming (e.g. soliciting and/or reporting the same $30 contribution in a different way), while others (lowering the number of classes with more than 20 students) may have a positive effect by themselves. But whenever the goal is one thing, but what is or can be measured is another, there of course will be incentives to respond to what is being measured.
Showing posts with label reputation. Show all posts
Showing posts with label reputation. Show all posts
Saturday, July 4, 2009
Friday, June 12, 2009
The Economic Logic of Anonymity and Pseudonymity
My previous post talks about my recent NBER working paper with Muriel Niederle and Utku Unver. Earlier this week I received an email that this paper was featured on another blog, Economic Logic. The email was signed "Economic Logician." And when I looked carefully at the blog itself (which I had read before; it's one of the economics blogs I have on my bookmark list), I couldn't find any identifying information about the blogger, even after clicking on the links through which blog owners often identify themselves.
So I replied to the email asking "Are you blogging anonymously, incidentally? (May I ask why?)"
I received the following reply: "Yes, I remain anonymous. This allows me more liberty in my writing, and I hope the blog will gather reputation on its own merit, instead of being attached to a name."
This got me thinking about the market design aspects of anonymity , and its not so close relative pseudonymity. When I asked EL about his/her anonymity, I was referring of course to the fact that I don't know who he/she is, what his/her day job is, which personal pronoun is appropriate, etc. But when EL speaks of the blog developing a reputation, he/she is referring to the fact that the posts are under the repeated-use pseudonym Economic Logician (with respect to which the name of the blog is an eponym), so that each post can both rely on and help establish the reputation of the same unidentified/pseudonymous person who wrote the previous posts.
Lots of internet markeplaces allow pseudonymous usernames, and still manage to foster reputation building and trust. (In an earlier post I wrote about the redesign of eBay's reputation system from one in which feedback could only be identified or pseudonymous to one in which it can also be anonymous: "ENGINEERING TRUST - RECIPROCITY IN THE PRODUCTION OF REPUTATION INFORMATION," - by Gary Bolton, Ben Greiner, and Axel Ockenfels. ) That paper notes how being able to identify who left feedback led to reciprocal feedback, either jointly positive or jointly negative, which reduced the informativeness of certain kinds of feedback.
There are other contexts in which anonymity is encouraged or allowed (and not just in comments to blogs). The refereeing of academic papers for publication is an example in which we seem to believe that the benefits (increased frankness) of anonymity may overcome the costs (e.g. an increase in self-serving reviews).
Some journals have double-blind refereeing, in which the authors as well as the referees may choose to remain anonymous during the review process (e.g. by deleting their names from the cover sheet, and also not posting the paper where Google can find it). Google now essentially prevents journals from enforcing anonymity on authors, since it gives authors a way to identify themselves if they wish, despite the requirement that their names and affiliations may not appear on the submitted paper. I think that preventing referees from knowing who the authors are would be a mixed blessing; it would level the playing field in certain desirable ways, but it might also remove information that would be valuable in evaluating the paper. (For example, if you read a paper saying that, on balance, the conclusions of Roth (1977) need to be revisited, it might be useful for you to know if Roth is one of the authors.)
Of course, marketplace rules may allow participants to be anonymous from some parties and not from others, and internet privacy rules are concerned with who can choose how individuals are identified. (See this nice take on the famous New Yorker cartoon about anonymity on the internet.)
So, anonymity has its uses, as does identity, and pseudonymity has some of the advantages and disadvantages of each. I'll keep blogging under my natural name for the time being, since, especially when my posts are unclear or incomplete, they may get some helpful context from the fact that they are linked to me and the work I do .
So I replied to the email asking "Are you blogging anonymously, incidentally? (May I ask why?)"
I received the following reply: "Yes, I remain anonymous. This allows me more liberty in my writing, and I hope the blog will gather reputation on its own merit, instead of being attached to a name."
This got me thinking about the market design aspects of anonymity , and its not so close relative pseudonymity. When I asked EL about his/her anonymity, I was referring of course to the fact that I don't know who he/she is, what his/her day job is, which personal pronoun is appropriate, etc. But when EL speaks of the blog developing a reputation, he/she is referring to the fact that the posts are under the repeated-use pseudonym Economic Logician (with respect to which the name of the blog is an eponym), so that each post can both rely on and help establish the reputation of the same unidentified/pseudonymous person who wrote the previous posts.
Lots of internet markeplaces allow pseudonymous usernames, and still manage to foster reputation building and trust. (In an earlier post I wrote about the redesign of eBay's reputation system from one in which feedback could only be identified or pseudonymous to one in which it can also be anonymous: "ENGINEERING TRUST - RECIPROCITY IN THE PRODUCTION OF REPUTATION INFORMATION," - by Gary Bolton, Ben Greiner, and Axel Ockenfels. ) That paper notes how being able to identify who left feedback led to reciprocal feedback, either jointly positive or jointly negative, which reduced the informativeness of certain kinds of feedback.
There are other contexts in which anonymity is encouraged or allowed (and not just in comments to blogs). The refereeing of academic papers for publication is an example in which we seem to believe that the benefits (increased frankness) of anonymity may overcome the costs (e.g. an increase in self-serving reviews).
Some journals have double-blind refereeing, in which the authors as well as the referees may choose to remain anonymous during the review process (e.g. by deleting their names from the cover sheet, and also not posting the paper where Google can find it). Google now essentially prevents journals from enforcing anonymity on authors, since it gives authors a way to identify themselves if they wish, despite the requirement that their names and affiliations may not appear on the submitted paper. I think that preventing referees from knowing who the authors are would be a mixed blessing; it would level the playing field in certain desirable ways, but it might also remove information that would be valuable in evaluating the paper. (For example, if you read a paper saying that, on balance, the conclusions of Roth (1977) need to be revisited, it might be useful for you to know if Roth is one of the authors.)
Of course, marketplace rules may allow participants to be anonymous from some parties and not from others, and internet privacy rules are concerned with who can choose how individuals are identified. (See this nice take on the famous New Yorker cartoon about anonymity on the internet.)
So, anonymity has its uses, as does identity, and pseudonymity has some of the advantages and disadvantages of each. I'll keep blogging under my natural name for the time being, since, especially when my posts are unclear or incomplete, they may get some helpful context from the fact that they are linked to me and the work I do .
Wednesday, May 27, 2009
Where did all the MD license plates go?
When I was a child, it was commonplace to see cars with license plates indicating that the car belonged to a medical doctor. Where did they all go?
It isn't that they aren't still available to doctors who want them, e.g. here is the MA application form, complete with a picture of an MD license plate: Application for Medical Doctor (MD) Plates . So why have doctors reduced their use of this kind of signaling?
My guess is the answer has to do with changes in markets that changed the value of sending such a signal. Here are some conjectures:
Housecalls play a much smaller role in medical practice than they did when I was a child, and so the need for docs to park in odd places and rely on their plates to ward off parking tickets has decreased.
The reputational benefits of being a doctor have decreased.
Drug addicts started to break into cars with MD plates, looking for drugs to use or sell.
Feel free to leave other conjectures as comments...
It isn't that they aren't still available to doctors who want them, e.g. here is the MA application form, complete with a picture of an MD license plate: Application for Medical Doctor (MD) Plates . So why have doctors reduced their use of this kind of signaling?
My guess is the answer has to do with changes in markets that changed the value of sending such a signal. Here are some conjectures:
Housecalls play a much smaller role in medical practice than they did when I was a child, and so the need for docs to park in odd places and rely on their plates to ward off parking tickets has decreased.
The reputational benefits of being a doctor have decreased.
Drug addicts started to break into cars with MD plates, looking for drugs to use or sell.
Feel free to leave other conjectures as comments...
Sunday, May 17, 2009
Piracy and armed merchantmen
Why is this headline such an outlier? Cruise ship opens fire to beat back Somali pirates
One reason is that ship owners are concerned about the longer term effects of escalating the level of violence in their encounters with pirates. But, it turns out, there are some market design reasons too (emphasis added):
"There have been calls for commercial ships to be allowed to carry weapons to deter increasingly bold pirate gangs, who are armed with automatic rifles and often rocket-propelled grenades.
But ships with arms onboard are not allowed to dock at non-military ports.
To bypass this rule, some operators are hiring private security teams who board as the ship enters a risky stretch of water and leave once the danger has passed.
Andrew Mwangura, head of the East African Seafarers' Assistance Programme in the Kenyan port of Mombasa, criticised the Melody for carrying guns.
"There are a number of other methods which can be used to deter the pirates, having weapons on board is dangerous because it raises the stakes for the pirates," he said.
"There is a far higher risk that a crew member of a merchant vessel, or a passenger, could die if the pirates feel they must fight harder to win the ship." The International Maritime Bureau said piracy incidents nearly doubled in the first quarter of 2009, almost all of them off Somalia. There were 18 attacks off the Somali coast in March alone.
Pirates have made millions of dollars from seizing ships and taking crews hostage. A Greek ship was released on Saturday after a £1.3 million ransom was paid.
But just hours earlier a German grain carrier was grabbed in the Gulf of Aden. "
One reason is that ship owners are concerned about the longer term effects of escalating the level of violence in their encounters with pirates. But, it turns out, there are some market design reasons too (emphasis added):
"There have been calls for commercial ships to be allowed to carry weapons to deter increasingly bold pirate gangs, who are armed with automatic rifles and often rocket-propelled grenades.
But ships with arms onboard are not allowed to dock at non-military ports.
To bypass this rule, some operators are hiring private security teams who board as the ship enters a risky stretch of water and leave once the danger has passed.
Andrew Mwangura, head of the East African Seafarers' Assistance Programme in the Kenyan port of Mombasa, criticised the Melody for carrying guns.
"There are a number of other methods which can be used to deter the pirates, having weapons on board is dangerous because it raises the stakes for the pirates," he said.
"There is a far higher risk that a crew member of a merchant vessel, or a passenger, could die if the pirates feel they must fight harder to win the ship." The International Maritime Bureau said piracy incidents nearly doubled in the first quarter of 2009, almost all of them off Somalia. There were 18 attacks off the Somali coast in March alone.
Pirates have made millions of dollars from seizing ships and taking crews hostage. A Greek ship was released on Saturday after a £1.3 million ransom was paid.
But just hours earlier a German grain carrier was grabbed in the Gulf of Aden. "
Saturday, April 4, 2009
More on recommender systems for escorts
Scott Cunningham of Baylor writes:
"I saw today your post on the screening sites that prostitutes use ["Problem customer" registries for prostitutes]. I just wanted to pass along to you a paper I'm currently working on with Todd Kendall called "Prostitution 2.0: Estimating the Effect of the Internet on the Market for Commercial Sex in the US."
http://business.baylor.edu/scott_cunningham/Research_files/pro20-4.pdf
"We have been surveying online prostitutes for the last 10 months using data collected from The Erotic Review, which is a large "mall" that clients use to record detailed information about prostitutes in various cities. In the course of the surveys, we have been studying the way in which the Internet has facilitated improved screening, and even in some cases, signaling between clients and prostitutes. One of the more ingenious things that we have found is a case of signaling from clients to prostitutes wherein they send letters of recommendation to prostitutes, usually in the form of sending along information [from] another prostitute with whom they've already visited. This, we argue, has enabled prostitutes to update their beliefs that a new client is not a cop (or a violent client) because these letters are relatively more expensive for cops to send. A case in point - see this story about two police officers who were sleeping with prostitutes in Beaumont, Texas allegedly in order to make a case on a drug trafficker. When the public learned, the men were fired, and the officers are now suing the Beaumont, Texas police department for wrongful termination. (It's suggestive that indeed the private costs of using these kinds of methods are prohibitively high for cops, which we argue is evidence that the costs of signaling type to prostitutes is such that the separating equilibria and not the pooling equilibria is more likely to be happening). "
Here is The Erotic Review, and here is what they say about customers getting recommendations from escorts:
"TER White List
If you have a good reputation with the ladies, encourage them to visit the TER White List and submit a referral for you. The TER White List is an easy way for providers to give positive references about members they have seen. "
"I saw today your post on the screening sites that prostitutes use ["Problem customer" registries for prostitutes]. I just wanted to pass along to you a paper I'm currently working on with Todd Kendall called "Prostitution 2.0: Estimating the Effect of the Internet on the Market for Commercial Sex in the US."
http://business.baylor.edu/scott_cunningham/Research_files/pro20-4.pdf
"We have been surveying online prostitutes for the last 10 months using data collected from The Erotic Review, which is a large "mall" that clients use to record detailed information about prostitutes in various cities. In the course of the surveys, we have been studying the way in which the Internet has facilitated improved screening, and even in some cases, signaling between clients and prostitutes. One of the more ingenious things that we have found is a case of signaling from clients to prostitutes wherein they send letters of recommendation to prostitutes, usually in the form of sending along information [from] another prostitute with whom they've already visited. This, we argue, has enabled prostitutes to update their beliefs that a new client is not a cop (or a violent client) because these letters are relatively more expensive for cops to send. A case in point - see this story about two police officers who were sleeping with prostitutes in Beaumont, Texas allegedly in order to make a case on a drug trafficker. When the public learned, the men were fired, and the officers are now suing the Beaumont, Texas police department for wrongful termination. (It's suggestive that indeed the private costs of using these kinds of methods are prohibitively high for cops, which we argue is evidence that the costs of signaling type to prostitutes is such that the separating equilibria and not the pooling equilibria is more likely to be happening). "
Here is The Erotic Review, and here is what they say about customers getting recommendations from escorts:
"TER White List
If you have a good reputation with the ladies, encourage them to visit the TER White List and submit a referral for you. The TER White List is an easy way for providers to give positive references about members they have seen. "
Monday, March 30, 2009
"Problem customer" registries for prostitutes
High end prostitutes and others who do business as "escorts" are vulnerable to booking bad customers, who may be abusive, fail to pay, be undercover police officers, insist on unsafe sex, or simply fail to keep their appointments. Because the services prostitutes sell are largely illegal, and because their customers may be anonymous, the market design problem of establishing a recommender/reputation feedback system to identify problem customers is considerable.
But there are several efforts in this direction, including sites which attempt to be available to the profession only (e.g. ProviderBuzz and DangerZone411), and, more accessibly, the National Blacklist Deadbeats Registry (Serving the Escorts Community)
"Our Vision: To see the day when female escorts (meaning; adult service providers, sex workers, call girls, courtesans, etc.) can work free from harm, and with peace of mind from pests, scammers, abusers, harassers, and stalkers."
The site (which requires a paid subscription to be able to search the reports) invites escorts to report and (try to) identify problem customers. The "incident report" form prompts the reporter for any available information identifying the customer (name, email address or phone number, "stated occupation," etc.), for a full physical description, and for a description of the incident, including an address (but with the warning "DO NOT put your address if the incident was at your incall location. You do not want to show YOUR address!")
Abbreviations for frequently reported categories are helpfully provided:
"Legend For Abbreviations
BC Boundary Crosser or Rule Breaker
BAYR Book At Your Risk
BBBJ Bare Back Blow Job
BBFS Bare Back Full Service
BBG Bare Back Greek (Anal)
DNS DO NOT SEE!!! "Bad Client Warning" (Various Reasons)
FDFK Forced Deep French Kissing
ILE Impersonating LE
LE Law Enforcement
M Manipulative, or Threatens to "out you" or ruin your working reputation
PST Phone or internet harasser/phone stalker
RAPE Rapist, any forced penetration without consent
ROB Refuse to pay or took back $ after
RCON Sneaks off condom
SC Short Changer
ST Stalker
NS No Show (NSNC - No Show, No Call)
STD Visible STD
V Violent
VP Physically violent or rough, assault, battery
VA Verbally abusive, rude, threatening, demeaning"
But there are several efforts in this direction, including sites which attempt to be available to the profession only (e.g. ProviderBuzz and DangerZone411), and, more accessibly, the National Blacklist Deadbeats Registry (Serving the Escorts Community)
"Our Vision: To see the day when female escorts (meaning; adult service providers, sex workers, call girls, courtesans, etc.) can work free from harm, and with peace of mind from pests, scammers, abusers, harassers, and stalkers."
The site (which requires a paid subscription to be able to search the reports) invites escorts to report and (try to) identify problem customers. The "incident report" form prompts the reporter for any available information identifying the customer (name, email address or phone number, "stated occupation," etc.), for a full physical description, and for a description of the incident, including an address (but with the warning "DO NOT put your address if the incident was at your incall location. You do not want to show YOUR address!")
Abbreviations for frequently reported categories are helpfully provided:
"Legend For Abbreviations
BC Boundary Crosser or Rule Breaker
BAYR Book At Your Risk
BBBJ Bare Back Blow Job
BBFS Bare Back Full Service
BBG Bare Back Greek (Anal)
DNS DO NOT SEE!!! "Bad Client Warning" (Various Reasons)
FDFK Forced Deep French Kissing
ILE Impersonating LE
LE Law Enforcement
M Manipulative, or Threatens to "out you" or ruin your working reputation
PST Phone or internet harasser/phone stalker
RAPE Rapist, any forced penetration without consent
ROB Refuse to pay or took back $ after
RCON Sneaks off condom
SC Short Changer
ST Stalker
NS No Show (NSNC - No Show, No Call)
STD Visible STD
V Violent
VP Physically violent or rough, assault, battery
VA Verbally abusive, rude, threatening, demeaning"
Thursday, March 26, 2009
Market for book reviews: Amazon version
Here's a nice description of how Amazon's reviews are ordered: at the top are the ones rated most useful, both positive and negative: The Magic Behind Amazon's 2.7 Billion Dollar Question
"Even the behavior of clicking Yes or No is elegant. Amazon tracks who rates each review as helpful, allowing each person to only vote once. This prevents "gaming the system" by voting for a friend's (or your own) review multiple times. Clicking either Yes or No pops up a quick message, saying the vote will take effect within 24 hours. (This delay also reduces gaming.)
Amazon quietly bumps the three most helpful reviews to the top. It tries to balance positive and negative reviews, so shoppers get a balanced perspective. An interesting side effect is how these selected reviews get more votes. If they are controversial (in that not everyone agrees they were helpful), their ratio goes down, allowing the most helpful reviews to bubble up past them."
And here is a discussion of how online recommendation systems might decrease the total diversity of products: Online Monoculture and the End of the Niche (HT to MR)
"Even the behavior of clicking Yes or No is elegant. Amazon tracks who rates each review as helpful, allowing each person to only vote once. This prevents "gaming the system" by voting for a friend's (or your own) review multiple times. Clicking either Yes or No pops up a quick message, saying the vote will take effect within 24 hours. (This delay also reduces gaming.)
Amazon quietly bumps the three most helpful reviews to the top. It tries to balance positive and negative reviews, so shoppers get a balanced perspective. An interesting side effect is how these selected reviews get more votes. If they are controversial (in that not everyone agrees they were helpful), their ratio goes down, allowing the most helpful reviews to bubble up past them."
And here is a discussion of how online recommendation systems might decrease the total diversity of products: Online Monoculture and the End of the Niche (HT to MR)
Thursday, March 5, 2009
Trust and trustworthiness: promoting and maintaining it
Trust is essential for all sorts of transactions, and how to set up institutions to promote trust and trustworthiness in the marketplace is a big concern of market design.
It is even a big concern of armies trying to win the hearts and minds of a population in the face of a guerrilla insurgency. There's a very interesting essay in the Washington Post about earning, maintaining and restoring trust, by Admiral Mike Mullen, the Chairman of the Joint Chiefs of Staff: Building Our Best Weapon.
On a more technical note, I'm reading Community Structure and Market Outcomes: Towards a Theory of Repeated Games in Networks by Itay Fainmesser (who you can try to hire next year). He is interested how patterns of connections between buyers and sellers can promote trustworthy behavior through repeated play, and how the effort to achieve trust in the marketplace by engaging in long term relationships may exclude some parties from the market. In this connection he writes "...repeated interactions cannot perfectly substitute for institutions..."
The kinds of institutions he is thinking of are both legal (if you can sue me for non-performance, this makes it easier to trust me), and reputational (if you can give me a credible negative review that will impede my ability to transact in the future, this also makes it easier to trust me).
As it happens, I've also been reading about the recent redesign of eBay's reputational system: "ENGINEERING TRUST - RECIPROCITY IN THE PRODUCTION OF REPUTATION INFORMATION," - by Gary Bolton, Ben Greiner, and Axel Ockenfels. It is a very nice market design paper.
They describe some of the design concerns behind eBay's 2007 rollout of its new, more detailed feedback system, in which buyers are able to give some feedback on sellers anonymously. In particular, they describe how they and eBay became concerned that the old reputation system became less informative than it might have been, because the pattern of reciprocally positive feedback concealed underlying dissatisfactions. They describe how (both prospectively and retrospectively) they compared the relevant field data, and how laboratory experiments helped verify the intuitions gained in that way, and allowed them to see the efficiency effects of an improved reputation system.
It is even a big concern of armies trying to win the hearts and minds of a population in the face of a guerrilla insurgency. There's a very interesting essay in the Washington Post about earning, maintaining and restoring trust, by Admiral Mike Mullen, the Chairman of the Joint Chiefs of Staff: Building Our Best Weapon.
On a more technical note, I'm reading Community Structure and Market Outcomes: Towards a Theory of Repeated Games in Networks by Itay Fainmesser (who you can try to hire next year). He is interested how patterns of connections between buyers and sellers can promote trustworthy behavior through repeated play, and how the effort to achieve trust in the marketplace by engaging in long term relationships may exclude some parties from the market. In this connection he writes "...repeated interactions cannot perfectly substitute for institutions..."
The kinds of institutions he is thinking of are both legal (if you can sue me for non-performance, this makes it easier to trust me), and reputational (if you can give me a credible negative review that will impede my ability to transact in the future, this also makes it easier to trust me).
As it happens, I've also been reading about the recent redesign of eBay's reputational system: "ENGINEERING TRUST - RECIPROCITY IN THE PRODUCTION OF REPUTATION INFORMATION," - by Gary Bolton, Ben Greiner, and Axel Ockenfels. It is a very nice market design paper.
They describe some of the design concerns behind eBay's 2007 rollout of its new, more detailed feedback system, in which buyers are able to give some feedback on sellers anonymously. In particular, they describe how they and eBay became concerned that the old reputation system became less informative than it might have been, because the pattern of reciprocally positive feedback concealed underlying dissatisfactions. They describe how (both prospectively and retrospectively) they compared the relevant field data, and how laboratory experiments helped verify the intuitions gained in that way, and allowed them to see the efficiency effects of an improved reputation system.
Subscribe to:
Posts (Atom)
Posts
