Saturday, July 4, 2009

Getting what you measure: college rankings version

As the rankings of universities conducted by the magazine US News and World Reports have become more influential, there are a growing number of reports of the ways, fair and not so fair, that universities respond to what USNWR tries to measure.

Clemson University has been in the news in connection with their stated efforts to rise higher in the US News and World Report rankings of colleges.
They and their critics agree that they want to do this; the question is are they doing it in the right way for the right reasons.

Here's a critic who says no:Researcher Offers Unusually Candid Description of University's Effort to Rise in Rankings:
"Clemson University is run in an almost single-minded direction, with nearly all policies driven by how they will help the land-grant institution rise in U.S. News & World Report’s rankings, according to a university official whose candid comments stirred debate among conference-goers here on Tuesday."

and the reply:
Clemson Assails Allegations That It Manipulates 'U.S. News' Rankings
"Clemson University, stung by charges by one of its own researchers that it willfully manipulates the U.S. News & World Report rankings, fired back on Wednesday, saying the accusations are “outrageous” examples of “urban legends” that have surrounded the university’s campaign to reach the top 20 of public research universities.“The accusation that Clemson, its staff, and administrators have engaged in unethical conduct to achieve a higher ranking is untrue and unfairly disparages the sincere, unwavering, and effective efforts of faculty and staff to improve academic quality over the past 10 years,” reads a statement issued by the university’s chief spokeswoman, Catherine T. Sams. “While we have publicly stated our goal of a top-20 ranking, we have repeatedly stressed that we use the criteria as indicators of quality improvement and view a ranking as the byproduct, not the objective.” "

Here's a summary: Clemson Explains Its Approach to U.S. News Rankings

And here's a story about alleged simple mis-counting at USC's School of Engineering: More Rankings Rigging , and a summary reflecting the relation between what is measured and what is reported: Gaming the Rankings. Here's an illuminating paragraph:

"Any performance measure is ripe to be gamed. The percentage of alumni giving is a measure worth 5 percent of a ranking in U.S. News. A few years ago, Albion College made its own stir in the higher education rankings world when it increased its percentage of alumni making donations with the stroke of a pen. As The Wall Street Journal reported, the college recorded a $30 donation from a graduating senior as a $6 alumnus gift for the next five years. Clemson, in its systematic approach to raising its rank — “no indicator, no method, no process off limits to create improvement,” as Watt stated — solicited alumni donations in such a way as to increase their giving rate: Alumni were encouraged to give as little as $5 annually."

Note incidentally that there are different ways to try to rise in the rankings, and some may be strictly gaming (e.g. soliciting and/or reporting the same $30 contribution in a different way), while others (lowering the number of classes with more than 20 students) may have a positive effect by themselves. But whenever the goal is one thing, but what is or can be measured is another, there of course will be incentives to respond to what is being measured.

No comments: