NYC school choice--cost overruns

David Chen in the NY Times has an article on cost overruns for the new school choice system which includes the high school choice system that Atila Abdulkadiroglu, Parag Pathak, and I helped design. (This is a good place to point out that we did our work pro bono--we didn't charge a penny. What we did get was the relevant choice data and the freedom to use it when we published papers based on our work.) More on our collaboration with NYC Dept of Ed, and our interaction with one of their software vendors, after the Times story, at which point I'll also draw some lessons about software vendors for school districts interested in designing better school choice systems. The article makes clear that the new high school choice system in NYC is viewed as a great success, but that the city had big problems with its software contractors.

Cost Overruns Found in Technology for Placement of High School Students

"For the last several years, the city’s Department of Education has boasted about its record of placing students in the high schools of their choice, thanks to a new computerized process. But if the ends were successful, the means were anything but, according to the city comptroller.

"In an audit released Thursday, the comptroller, John C. Liu, criticized the department’s handling of the finances of the computer system, which is modeled after a system that matches medical school graduates with residency programs.

"Instead of adhering to its original contract of $3.6 million for the system’s development, the audit found, the department spent $13.5 million. Then the original project was deemed insufficient and it had to be scrapped and the city had to spend an additional $9.4 million — and counting — for a new system.

"The gulf of more than $19 million between expected and projected costs reflected “poor planning” and raised concerns, the comptroller’s office said, about “similar cost overruns” for all technology projects.

"The city’s original contract was with Spherion, a company whose reputation was sullied in December when several of its consultants were indicted by federal prosecutors. The consultants were accused of concocting an $80 million corruption scheme related to another technological initiative, an automated payroll system called CityTime. The city hired a new vendor for the Education Department in 2008.

"In a letter to Cathleen P. Black, the schools chancellor, H. Tina Kim, the city’s deputy comptroller for audits, wrote, “Clearly, savings could have been achieved with better planning and coordination.”
...
"In response to the audit, city officials said that fast-moving changes in facets of education had necessitated upgrades — at additional costs — to the computer system. The program had to accommodate middle-school choices, prekindergarten admissions and programs for gifted elementary-school children.

"Jack Zarin-Rosenfeld, a spokesman for the Education Department, said: “Before this administration, we had a high school admissions system that was rife with political patronage and too often gamed by the well-connected, leaving students with the most need behind. Thanks to our policy changes and investments, we have greatly expanded equity and choice for all of our families, and provided more high-quality opportunities on a scale larger than any other city in the country.”

"With the advent of mayoral control of the schools shortly after Mr. Bloomberg took office in 2002, the Education Department overhauled the nerve-wracking and lotterylike process by which students were admitted to high schools. In the past, students could select only five schools, but some students were admitted to more than one, leaving the city’s best students to mull over their options after applying. But under the new High School Application Processing System, students are allowed to list 12 choices, and are matched to just one school.

"For the 2010-11 school year, 52 percent of the roughly 80,000 high-school-bound students who applied were matched to their first choices; in 2004, by contrast, that figure was 33.6 percent.

"In 2005, Joel I. Klein, then the schools chancellor, praised the new system, saying “this process, frankly, was a long time quite broken.”

"The original contract was amended eight times between 2003 and 2008, adding almost $10 million in costs, chiefly to adjust to changes in school policies. And those amendments, according to Ms. Kim’s letter, “reflect poor planning” and “appear to be an admission that the changes that occurred in D.O.E. were not expected or considered” in developing the new system.

"By 2008, the city had concluded that it needed a new vendor to develop a new and more flexible technology system, and hired Vanguard Direct Inc., under a contract expiring in 2013. "

*********************************************

The story reminds me of how in 2003, when I was first contacted by Jeremy Lack at the New York City Department of Education and asked to design a choice process, the city had already contracted with Spherion to be its software provider. It quickly became apparent that Spherion's core competence was getting contracts with the city. There were a number of indications that they were going to be difficult to work with. Two incidents stand out in my memory.

First, Spherion had prepared a plan that called for programming lots of modules to knit together different aspects of school choice. I recall a conference call in which several Spherion people participated together with Jeremy Lack, me and Parag Pathak, in which Parag and I pointed out that the (deferred acceptance algorithm) architecture we proposed would require vastly less programming to accomplish than the original contract allowed for. One of the Spherion people pointed out that, in that case, a contract modification would have to be negotiated, at additional cost to the city.

Second, the way the project progressed is that, as we agreed on particular design elements, Parag and Atila would each program a version (in Matlab and Perl, respectively) that they would check against each other by running test problems, and the algorithm would be passed on to Spherion. At one point we asked Spherion how they were planning to check that their programming was accurate, and they replied that checking the software wasn't called for in the contract (and, besides, they explained, there was a logical inconsistency in the notion that you could check the accuracy of software, since you would have to design software to do that, and that software would have to be checked...). So Parag and Atila did the software checking too.

Of course we programmed only the fun stuff, the choice algorithm, and a delivered system has to have lots of data entry and processing software, so that preference lists can be entered by middle school guidance counselors, etc. Still, I'm staggered by the cost figures reported by the Times. It's hard to imagine that a gold plated system, with 100% profit margins for the contractors, couldn't have come in at 10% of the reported cost.

Part of the problem, I suspect, is that the contract with Spherion didn't give the software to NYC, but left it with Spherion which was supposed to run the match, which I think increased charges each year. When we subsequently helped Boston redesign its school choice algorithm (with Tayfun Sonmez now also playing a critical role on the team), we worked in a similar way, but Boston ended up owning the software and running the match itself. This is a better financial/ownership model.

Neil Dorosin, the NYC DOE official who had the hands-on responsibility for implementing the new high school match, subsequently left the DOE to form a non profit institute to help school districts implement choice plans, the Institute for Innovation in Public School Choice (IIPSC). Atila, Parag and I are on his Board. Building on his considerable NYC experience, Neil probably now knows as much as anyone about the details of building a school choice system, and how the parts fit together, based on his experience not only in NY but in school districts around the country with whom he's been in contact.


Here's a paper in which the NYC high school choice system is described in detail, along with some details of its performance:
Abdulkadiroglu, Atila , Parag A. Pathak, and Alvin E. Roth, "Strategy-proofness versus Efficiency in Matching with Indifferences: Redesigning the NYC High School Match,'' American Economic Review, 99, 5, Dec. 2009, pp1954-1978. And here are the AER links at which you can access the Appendix and Download the Data Software