Friday, September 2, 2011

New York City Halts Teacher Bonus Program: Another Blow to Evidence-Resistant Ideology - Bob Sutton

New York City Halts Teacher Bonus Program: Another Blow to Evidence-Resistant Ideology - Bob Sutton

New York City Halts Teacher Bonus Program: Another Blow to Evidence-Resistant Ideology

The New York Times reports that the school system has abandoned their teacher bonus system because it is ineffective. I quote:

A New York City program that distributed $56 million in performance bonuses to teachers and other school staff members over the last three years will be permanently discontinued, the city Department of Education said on Sunday. The decision was made in light of a study that found the bonuses had no positive effect on either student performance or teachers’ attitudes toward their jobs.

The research appears to be quite careful and the RAND Corporation is highly respected:

The study, commissioned by the city, is to be published Monday by the RAND Corporation, the public policy research institution. It compared the performance of the approximately 200 city schools that participated in the bonus program with that of a control group of schools. Weighing surveys, interviews and statistics, the study found that the bonus program had no effect on students’ test scores, on grades on the city’s controversial A to F school report cards, or on the way teachers did their jobs. “We did not find improvements in student achievement at any of the grade levels,” said Julie A. Marsh, the report’s lead researcher and a visiting professor at the University of Southern California. “A lot of the principals and teachers saw the bonuses as a recognition and reward, as icing on the cake. But it’s not necessarily something that motivated them to change.”

Are you surprised? I am not, and if the people running the New York City school system had actually read a large body of existing research, they would never have wasted all this money in the first place. In our opening chapter of Hard Facts, Dangerous Half-Truths, and Total Nonsense, Jeff Pfeffer and I reviewed the extensive literature on the links between incentives and teacher performance, and it turns out that although there always have been people with great faith in pay for performance systems for teachers -- going back to at least 1918 -- careful studies show over and over again that they do not improve student performance. The New York Times article suggests that despite the ideology supporting pay for performance systems, there is growing evidence that the current round of incentive-based teacher pay isn't working -- just as it never had worked:

The results add to a growing body of evidence nationally that so-called pay-for-performance bonuses for teachers that consist only of financial incentives have no effect on student achievement, the researchers wrote. Even so, federal education policy champions the concept, and spending on performance-based pay for teachers grew to $439 million nationally last year from $99 million in 2006, the study said.

To be clear, pay for performance schemes do appear to have some effects in schools -- most of which are bad. One of the most well-documented (see this post on findings inFreakomomics and related research) is that some teachers and administrators start cheating when their pay is linked to performance on student's standardized tests. Their are strong hints that this is exactly what happened in Washington, D.C. and other cities where financial incentives for teachers and administrators are linked to student test scores.

Note that I am not arguing against pay for performance systems in general. They work in other settings --sports, sales, lots of other places,as we show in Hard Facts. But they don't work for teachers for a host of reasons, perhaps paramount among them are that teachers rarely have enough control over key student behaviors before, during, and outside of class, over class composition (and when they do, they sometimes use it to cheat the tests.. such as by sending poor perfomers to special education classes), and over other resources they need to have a strong enough impact on student learning. Also, giving students a test once a year probably isn't a very good way to measure what students are learning. As The New York Times report argues, another problem with pay for performance schemes is that it turns teachers' attention away from intrinsic rewards (the reason most go into the profession in the first place) and toward extrinsic rewards (See Dan Pink's Drive to learn more about the trouble with extrinsic rewards).

To be clear, I am NOT a general supporter of the policies of teacher's unions. Although I do think that way too much blame and way too little credit is given to teachers, I do have an evidence-based pet peeve against how vehemently teacher's union's defend the jobs of bad apples, the rotten and incompetent teachers. This argument is consistent with the work on "Bad is stronger than good" that I've discussed here before... while it is tough for even a good teacher to overcome a lousy system and have strong positive impact on students, it is pretty clear that really lousy teachers can make a bad system worse, and dampen the positive effects of a good one. I believe that if unions changed policy here and became even more vehement about reforming and removing bad teachers than their critics it would improve their reputations and the quality of education -- and earn them political capital to battle lousy policies such as tying teacher pay to student test performance. (See this great discussion and debate at The New York Times).

To return the dismal record of pay for performance systems in schools, some years back, I had an interesting conversation with Tony Bryk, a prestigious educational researcher who is now heads the Carnegie Foundation. We were were at a think tank, a place called the Center for Advanced Study in the Behavioral Sciences, and I asked Tony why -- even though there is so much evidence against practices like pay for performance for teachers -- they remain popular and come back in waves... until overwhelming evidence emerges again that in fact they are bad. Tony suggested two reasons. The first has to do with ideology -- that people hold some assumptions so strongly (like economics and business minded folks who believe that incentives are the best answer to changing any kind of human behavior) that they refuse to accept any evidence that runs counter to their beliefs -- no matter how strong those findings might be. The second reason was what Tony called "collective amnesia." He argued that, in the history of educational policy, the same bad ideas seem to come around every 10 or 20 years, and policy makers and their staffs either don't remember or make no effort to dig-up relevant research to guide their decisions, regardless of their ideologies. In the case of pay for performance, it appears that both of these factors are operating.

Practicing evidence-based management isn't easy given our various human flaws. But we sure could save a lot of money, a lot of heartache, and make people's lives a lot better if we all tried a lot harder to do it. There are plenty of outcomes in life that are impossible to predict. Unfortunately, what happened in New York was completely predictable, even if people with blind faith in linking test scores to financial rewards for schools and teachers remain unwilling to believe this well-established truth now.

P.S. A comment below suggests that some recent studies do show a positive effect of financial rewards on student performance. My reading of the Rand Report suggests that for studies done in the U.S. there are a a few studies that show a positive impact, but the weight of the evidence supports the historical pattern of no effects or negative effects. The more rigorous studies in particular find no ot weak effects on test scores, and little effect on teacher motivation, although there is some evidence that teachers devote more evidence to teaching to the test and less to teaching other things (not a surprise). There is also some evidence that teacher cooperation goes down a bit and evidence that teachers game the system more to boost test scores. A researcher from Chicago explained to me that in the schools she was studying (this was about 10 years ago) that scores were going up but she believed that it was not so much because incentives motivated teachers to work harder, but because it motivated them to get rid of their weakest students (often by sending them to special ed classes) and to refuse to "skip" gifted students because they pumped-up the average test scores in a class. Finally, the most obvious effects of pressure on teachers and administrators to pump up test scores is cheating on the tests (by the teachers and administrators), as we have seen from evidence from Chicago, Philadelphia, and Atlanta, and just today, a probe started into cheating at schools in New Jersey.

I am not rejecting the value of financial rewards as motivational tools for teachers outright, and it does appear that there are some special conditions under which they may be of some value. But the weight of the evidence suggests that most of the money spent on such incentives could have been to better use, that the ideological support for them is much stronger than than the evidence in support, and that one of the most consistent effects is bad -- teachers and administrators cheat either to get the incentives or because they fear losing their jobs.

This is my conclusion. You may reach a different one. Here is a link to the 300 page Rand study of the failed New York program, which contains an excellent and very current review of the research.

No comments:

Post a Comment