Cheating on exams in India , and exam design suggestions from Spain

 India, where government jobs are allocated based on scores on a variety of national exams, cheating is a big business. One form it takes is sale of upcoming exam questions. 

An experiment conducted in Spain by Klijn, Alaoui, and Vorsatz, which introduced multiple versions of an online exam, suggests that this may reduce cheating by people who take the exam after others have already taken it.

 From the NYT:

These Exams Mean Everything in India. Thieves See a Gold Mine.
In a country where government jobs are highly coveted, the tests that govern hiring are a lucrative target for criminal gangs. By Mujib Mashal and Hari Kumar

 "Allotting jobs on the basis of exam results conveys a sense of fairness. But with competition so fierce, the temptation to seek shortcuts can be strong.

"Some aspirants, while spending long hours in study groups, also keep an eye out for shadowy figures offering access to exams.

 

############

From the Indian Express

Bill in Lok Sabha to check paper leaks, use of unfair means in govt recruitment exams
At present, there is no specific substantive law to deal with unfair means adopted or offences committed by various entities involved in the conduct of public examinations by the central government and its agencies. 

"The Public Examinations (Prevention of Unfair Means) Bill, 2024, introduced by Union Minister of State for Personnel Jitendra Singh, mentions “leakage of question paper or answer key”, “directly or indirectly assisting the candidate in any manner unauthorisedly in the public examination” and “tampering with the computer network or a computer resource or a computer system” as offences done by a person, group of persons or institutions."

####### 

 And here's an experiment with exam design, involving a Spanish exam:

Online Academic Exams: Does Multiplicity of Exam Versions Mitigate Cheating?  by Flip Klijn,  Mehdi Mdaghri Alaoui, and Marc Vorsatz

Abstract: We study academic integrity in a final exam of a game theory course with 463 undergraduate students at a major Spanish university. The exam is an unproctored online multiple-choice exam without backtracking. A key characteristic is that for each (type of) problem, groups of students receive different versions. Moreover, each problem version is assigned to one subgroup during one stage of the exam and to another subgroup during an immediately consecutive later stage. Thus, we can exploit grade points and timestamps to study students’ academic integrity. We observe a significant decrease in completion time at each later stage; however, surprisingly, there is no corresponding impact on average grade points. The precise number of different versions does not seem to have an effect on either variable. Our findings thus suggest that employing a limited number of distinct problem versions (as few as two) can diminish cheating effectiveness in online exams."

Standardized testing returns to college admissions

 Standardized admissions exams are making a comeback:

Elite French university revives entrance exam to combat AI fears. Sciences Po reverses changes intended to make admissions more egalitarian, saying ChatGPT and private tutoring mean essays are no longer ‘objective’  by Maurício Alencar, Wednesday October 23 2024, 12.00pm BST, The Times 

"Sciences Po’s new director, Luis Vassy, said standardised testing would make admissions more “objective”, despite one of his predecessors, Frédéric Mion, dropping the exam in an attempt to make the process more accessible to poorer students who had to pay for travel into Paris to sit the test."

%%%%%%%%%%%

Here's the story from the Harvard Gazzette (in April):

Harvard announces return to required testing. Leading researchers cite strong evidence that testing expands opportunity

"Students applying to Harvard College for fall 2025 admission will be required to submit standardized test scores, the Faculty of Arts and Sciences announced ... This new policy will be applied to the Class of 2029 admissions cycle and will be formally assessed at regular intervals. 

...

"“Standardized tests are a means for all students, regardless of their background and life experience, to provide information that is predictive of success in college and beyond,” she said. “Indeed, when students have the option of not submitting their test scores, they may choose to withhold information that, when interpreted by the admissions committee in the context of the local norms of their school, could have potentially helped their application. In short, more information, especially such strongly predictive information, is valuable for identifying talent from across the socioeconomic range.”

 

"In research published last year, Harvard Professors Raj Chetty and David J. Deming and co-author John N. Friedman used data from more than 400 institutions and about 3.5 million undergrads per year to better understand socioeconomic diversity and admissions. Standardized tests emerged as an important tool to identify promising students at less-well-resourced high schools, particularly when paired with other academic credentials.

 

“Critics correctly note that standardized tests are not an unbiased measure of students’ qualifications, as students from higher-income families often have greater access to test prep and other resources,” said Chetty, the William A. Ackman Professor of Public Economics and director of Opportunity Insights. “But the data reveal that other measures — recommendation letters, extracurriculars, essays — are even more prone to such biases. Considering standardized test scores is likely to make the admissions process at Harvard more meritocratic while increasing socioeconomic diversity."

Return to previous school assignment policies (in some respects) under New York City's new mayor

 In NYC, the pendulum is still swinging between inclusive admissions as measured by demographics and determined by lottery, and meritocratic admissions as measured by tests and grades.

The NYT has the story:

In a Reversal, New York City Tightens Admissions to Some Top Schools. The city loosened selection criteria during the pandemic, policies some parents protested as unfair and others hoped would reduce racial disparities. By Troy Closson

"New York City’s selective middle schools can once again use grades to choose which students to admit, the school chancellor, David C. Banks, announced on Thursday, rolling back a pandemic-era moratorium that had opened the doors of some of the city’s most elite schools to more low-income students.

...

"New York City has used selective admissions for public schools more than any school district in the country. About a third of the city’s 900 or so middle and high schools had some kind of admissions requirement before the pandemic disrupted many measures to sort students by academic performance.

...

"Selective high schools will also be able to prioritize top-performing students.

"The sweeping move will end the random lottery for middle schools, a major shift after the previous administration ended the use of grades and test scores two years ago. At the city’s competitive high schools, where changes widened the pool of eligible applicants, priority for seats will be limited to top students whose grades are an A average.

...

"The announcement came as New York City’s education officials are confronting multiple crises in the wake of the pandemic, complicating a dilemma that has bedeviled previous administrations: how to create more equitable schools, while trying to prevent middle-class families from abandoning the system.

"State standardized test scores released Wednesday showed that many students fell behind, particularly in math, and that many Hispanic, Black and low-income students continue to lag far behind their white, Asian and higher-income peers. At the same time, the district is bleeding students: Roughly 120,000 families have left traditional public schools over the past five years. Some have left the system, and others have gone to charter schools."

*****

And here's the Washington Post:

New York City, embracing merit, rolls back diversity plan for schools By Laura Meckler

"New York City schools announced Thursday they would allow middle schools to consider academics in admitting students to some of the city’s most sought-after programs, unraveling pandemic-era rules aimed at injecting racial and economic diversity into a segregated system.

"High schools would also rely more heavily on merit and less on the luck of a lottery under the new plan, reversing the previous administration’s direction as a new mayor takes command of the nation’s largest school system.

...

"In San Francisco, admissions into the elite Lowell High School were converted from merit-based into a lottery system. As in New York, though, the change was reversed — in this case, after several school board members were recalled, in part over this issue.

"In Northern Virginia, Thomas Jefferson High School for Science and Technology also shifted from an admissions test to a “holistic review” that considers several factors, a move that is being challenged in court and has faced resistance from the Republican governor and his administration.

...

"In New York, the debate is particularly fiery because students are required to apply to middle and high school, and before the pandemic, about a third of the city’s 900 middle and high schools included requirements for admission — such as grades, test scores, attendance and behavior records. 

...

"That system was largely converted into a lottery under Mayor Bill de Blasio.

"For high school, applicants were put into tiers based on their grades. But the top tier included about 60 percent of all students, who had the first crack at the top schools. Competitive schools drew acceptances randomly from this group.

...

"Now, under the new system announced Thursday, it will be harder to get into the top tier, though once in that group, it will still be a lottery. To get into the top tier, students must be in the top 15 percent of their school or of the city overall, and they must have at least a 90 percent on grades.

"Test scores, which had been used for years but also criticized as biased, will not be considered. Banks said exam scores are a flawed measure but grades are “still a very solid indicator of how you are showing up as a student,” even for students who face hardships at home."

Reducing the costs of preparing for high stakes exams by reporting scores coarsely

 In many countries, national exams serve as the gateway to college admissions and other prizes, and many applicants incur great costs in time and treasure preparing for these exams.  Here's a recent NBER working paper that suggests that reporting the grades in intervals rather than by individual scores has the potential to reduce the costs devoted to exam prep sufficiently to be a Pareto improvement for students, i.e. to make them all better off, even those who obtain the highest grades, if the cost of doing so is sufficiently high.

Pareto Improvements in the Contest for College Admissions by Kala Krishna, Sergey Lychagin, Wojciech Olszewski, Ron Siegel & Chloe Tergiman, NBER WORKING PAPER 30220, DOI 10.3386/w30220, July 2022

Abstract: "College admissions in many countries are based on a centrally administered test. Applicants invest a great deal of resources to improve their performance on the test, and there is growing concern about the large costs associated with these activities. We consider modifying such tests by introducing performance-disclosure policies that pool intervals of performance rankings, and investigate how such policies can improve students’ welfare in a Pareto sense. Pooling affects the equilibrium allocation of studentso colleges, which hurts some students and benefits others, but also affects the effort students exert. We characterize the Pareto frontier of Pareto improving policies, and also identify improvements that are robust to the distribution of college seats.

"We illustrate the potential applicability of our results with an empirical estimation that uses data on college admissions in Turkey. We find that a policy that pools a large fraction of the lowest performing students leads to a Pareto improvement in a contest based on the estimated parameters. We then conduct a laboratory experiment based on the estimated parameters to examine the effect of such pooling on subjects’ behavior. The findings generally support our theoretical predictions. Our work suggests that identifying and introducing Pareto improving performance-disclosure policies may be a feasible and practical way to improve college admissions based on centralized tests."

The paper notes that:

" In many Asian countries, including China, Japan, South Korea, and Taiwan, students attend specialized “cram schools,”1 which focus on improving students’ performance on the tests. This often consists of rote learning, solving a large number of practice problems, and practicing test-taking strategies tailored to the specific test. In other countries, students hire tutors, buy books, and take specialized courses, all geared entirely toward improving their test scores. These activities likely improve students’ performance on the test, but are far less likely to generate substantial long-term improvements in students’ productive human capital. These activities do, however, carry significant costs in terms of time, money, and effort. In South Korea, for example, it is not uncommon for high school students to spend several hours a day in cram schools, and the high stakes competition for college admissions is seen as one of the main causes for the high rates of unhappiness and suicide among teenagers.2 Similar concerns have also been raised in the United States.3"

The paper explains that:

"We are interested in performance-disclosure policies that benefit all students, and refer to such policies as Pareto improving. In particular, we do not need to consider welfare tradeoffs across students. A key finding of our analysis is that Pareto improving policies often exist. This may seem surprising, since a fixed set of college seats implies that a student can be admitted to a better college only if another student is admitted to a worse college. The crucial element that makes Pareto improvements possible is that test preparation is costly. The costs students incur, as well as the resulting college assignment, are determined in equilibrium, and the equilibrium is affected by the performance-disclosure policy. Relative to the baseline contest with no coarsening, introducing a performance-disclosure policy leads to some students being admitted to better colleges; this makes them better off even if they incur higher costs, as long as the cost increase is not too large. Other students are admitted to worse colleges; if they also incur lower costs they are made better off as long as the reduction in the costs is large enough."

**********

I'm reminded of a paper that suggests that the very best students may not pay the highest costs for exam prep:

Feltovich, Nick, Richmond Harbaugh, and Ted To. "Too cool for school? Signalling and countersignalling." RAND Journal of Economics (2002): 630-649.

San Francisco's Lowell High School admissions will return to merit-based system

 The SF Chronicle has the latest twist in this involved story over San Francisco's elite Lowell High School.

Lowell High School admissions will return to merit-based system after S.F. school board vote  by Jill Tucker

"After nearly two years of intense and bitter debate, test scores and grades will once again determine which San Francisco students are admitted to Lowell High School after the city’s school board decided to return to the merit-based admission system Wednesday.

"In a 4-3 vote, the school board decided to restore the previous merit process after two years of using a lottery-based system. The vote will now apply to freshman entering in the fall of 2023 as well as future classes, unless the board takes further action in the future to change the admission process.

...

"The board’s decision was the latest inflection point in the nearly two-year saga featuring feuding public officials, a lawsuit and accusations of racism over which students are eligible to attend Lowell, long considered one of the highest-performing public high schools in the country.

"The board first approved a switch to a lottery system in October 2020, citing a lack of academic data given the switch to distance learning earlier that year.

"A board majority then made that decision permanent four months later, citing a lack of diversity and racism at the elite academic schools. But the hurried vote sparked a lawsuit and then a judge’s ruling that the district violated laws related to the Brown Act, which regulate public meetings.

"The board then had to backpedal, reversing the decision before extending the lottery process for another year."

***********

Earlier:

Sunday, April 17, 2022

Lowell High School principal resigns in San Francisco, in latest setback for elite high school

Who Benefits from Meritocracy? by Diana Moreira & Santiago Pérez

 Exams for U.S. civil service positions apparently started for some positions in 1883, and here's an NBER working paper that looks at the difference that made in the composition of people hired, by socioeconomic status.

Who Benefits from Meritocracy?  by Diana Moreira & Santiago Pérez

NBER WORKING PAPER 30113 DOI 10.3386/w30113 June 2022

Does screening applicants using exams help or hurt the chances of lower-SES candidates? Because individuals from lower socioeconomic backgrounds fare, on average, worse than those from richer backgrounds in standardized tests, a common concern with this "meritocratic" approach is that it might have a negative impact on the opportunities of lower-SES individuals. However, an alternative view is that, even if such applicants underperformed on exams, other (potentially more discretionary and less impersonal) selection criteria might put them at an even worse disadvantage. We investigate this question using evidence from the 1883 Pendleton Act, a landmark reform in American history which introduced competitive exams to select certain federal employees. Using newly assembled data on the socioeconomic backgrounds of government employees and a difference-in-differences strategy, we find that, although the reform increased the representation of "educated outsiders" (individuals with high education but limited connections), it reduced the share of lower-SES individuals. This decline was driven by a higher representation of the middle class, with little change in the representation of upper-class applicants. The drop in the representation of lower-SES workers was stronger among applicants from states with more unequal access to schooling as well as in offices that relied more heavily on connections prior to the reform. These findings suggest that, although using exams could help select more qualified candidates, these improvements can come with the cost of increased elitism.

From the conclusions:

"Our findings have implications for the broader debate on exams and meritocracy. Allocating opportunities based on exams is sometimes described as an equity-efficiency panacea, helping select the most qualified candidates while simultaneously increasing the representation of lower SES individuals. Our results challenge this view: although using exams could, in principle, help select more qualified candidates, we show that these improvements can also come with the cost of increased elitism. More generally, our findings show that adopting less discretionary selection criteria might not necessarily help the chances of lower-SES individuals.

...

"Importantly, while we investigate how exams shaped the social origins of government officials, an important question that remains unanswered is whether the poor themselves were on net made worse off by the reform. The answer to this question is not an obvious one for a variety of reasons. For instance, individuals from disadvantaged backgrounds might benefit the most from having a well-functioning state, even if achieving this efficiency implies that they might lose direct access to government jobs. "

***

This paper makes me think of an earlier paper, about the historical introduction and then abandonment of a national exam-based school choice system in Japan, where the result of national exams was that urban students filled more of the places...

Friday, February 21, 2020

School choice, centralized and decentralized, in Japan, a century ago, by Tanaka, Narita and Moriguchi

What are (swim) tests for?

 Tests serve multiple purposes, they can be to screen, to certify, to incentivize.  In recent years we have seen concern about tests' differential predictive power for different groups of test takers lead to reduced use for screening, e.g. as SAT's have become optional for college admissions, or as the US Medical Licensing Exam (USMLE) step 1 is moving to pass/fail grades, to prevent its (over)use in screening medical students for residencies.

Here's a story in the Chronicle of Higher Ed on swimming tests being abandoned at colleges, most recently at Williams.  (It brought back memories of my own freshman swim test at Columbia in 1968... )

Race on Campus: Why Colleges Are Dropping Their Swim Tests by Adrienne Lu 

"Welcome to Race on Campus. At one point or another, about a quarter of American colleges required students to pass a swim test. Today, there are far fewer who do, but a few holdouts remain. One college dropped its requirement this month after examining data showing students of color were far more likely to need a remedial swim class. 

...

“Students expressed feeling shamed and punished for not knowing how to swim,” wrote D. Clinton Williams in an email to The Chronicle. Williams is director of the college’s Pathways for Inclusive Excellence and chairman of the Diversity Advisory Research Team, which studied the swimming requirement. “One student told her first-year adviser, ‘It’s like they are punishing the city kids.’

...

"Once common among American colleges, swimming requirements have been dwindling for decades. A 1997 survey by professors at North Carolina State University found only 5 percent of colleges had a swim-test requirement then, down from the 25 percent that once did.

...

"Concerns about how swim requirements affect students differently are not new. Hobart and William Smith Colleges dropped its requirement in 1994, calling it archaic, difficult to administer, and unfair to students who had no access to pools, according to a Chronicle article at the time. An op-ed in the student newspaper of Washington and Lee University last November argued that the university had “failed to consider racial, economic, and cultural barriers to swimming.”

"Jeff Wiltse, a professor of history at the University of Montana, who has published extensively on the history of swimming pools in the U.S., said Black Americans today are about half as likely to know how to swim as white Americans.

...

"According to the Centers for Disease Control and Prevention, drowning is one of the three leading causes of unintentional-injury death among Americans under 29 years old, and from 1999-2019, American Indian or Alaskan Native people died from drowning at twice the rate of non-Hispanic white people, while non-Hispanic Black people died from drowning at 1.5 times the rate."

*********

The Talmud (in a section that's not so easy to read, kiddushin 29a, concerned among other things with a father's obligations to a son) records this opinion: " And some say: A father is also obligated to teach his son to swim. "