4 ms·

OK, so they analyzed 69,000 schools by comparing test scores to a model they created predicting what the scores should be, and 200 out of the 69,000, or 0.3%, d

OK, so they analyzed 69,000 schools by comparing test scores to a model they created predicting what the scores should be, and 200 out of the 69,000, or 0.3%, didn't match the model. (Note: main article says 200 districts, but the associated article showing the data says 200 schools http://www.ajc.com/news/cheating-our-children-1393866.html http://www.ajc.com/news/cheating-our-children-1393866.html) It's amazing that 99.7% of schools matched their model. Must be a very good model. It's ridiculous that they are using this to say there is evidence of cheating though since, even though their model is 99.7% accurate, it is not 100% accurate, and that's to be expected. The methodololgy also relies on questionable assumptions. They look at 3 year slices to find cases where test scores jumped up one year and then back down the next. They consider these evidence of systematic, district wide cheating. That is unproven. If it was cheating, why would the scores go back down. Is there a correlation then between years of upswing and schools being on the list of underperforming schools about to be forcibly shut down by the feds? That might be evidence of cheating if so. But if that was so it would surely be claimed in the study. Also, district wide cheating means a mass conspiracy in which hundreds of teachers managed to, under direction of a coordinated central authority, successfully cheat for one year but not the next, and to keep the secret, and did so at hundreds of schools. That's quite a conspiracy. I am sure cheating takes place from time to time, but it is far more likely to happen at the individual teacher level than as something happening district wide and thus coming from a central source because the larger a conspiracy gets, the less likely it is that everyone can keep the secret. Checking for schools that have a single year with higher scores and finding a small number that do is poor evidence of a cheating conspiracy.

tokenadult 15y ago
I appreciate your thoughtful comments, droithomme, on many of the HN threads on education reform. You are correct that the statistical model used by the newspaper report needs more validation. On the other hand, as the story relates, the model is a generalization of a model developed during reporting by the same newspaper about known cheating incidents that could be independently verified by other means, including witness reports of overt conduct that breached test security or altered student-submitted answers after the test booklets were submitted to teachers or school principals. The earlier reporting by the same newspaper revealed that sometimes agreements ("conspiracies") to cheat on tests were rather widespread in entire schools or even entire school districts, and stayed secret for a time (at least) because all the school officials involved were united in having an interest in presenting fake test scores. I think that the scores bouncing up but then back down is, again, an empirical observation from the known cases of cheating in the area the newspaper reported on most closely a few years ago. As the submitted article here suggests, it is certainly notionally possible that a group of students might spurt forward suddenly in academic performance if they enjoy the instruction of a particularly good teacher. But, if genuine learning had taken place, it is notionally much harder to find the same group of students fall back the next year to a level lower than the level they reached the previous school year. It is much more likely that students might progress in grades from a teacher who is eager to look good, even if it means cheating, to a teacher who is not in on the cheating plot, or who at least is less effective at cheating in a manner that massively boosts scores. I think some of the other claims that you suggest would be made by the newspaper, if the statistical model were definitely backed up by other evidence of cheating, could be made if journalists in the various localities involved look for other sources of information. Not in all cases, as you correctly point out, but in many cases, additional reporting leg work could probably find direct evidence of violation of test security before the test resulting in exact "teaching to the test" item by item, or alteration of student item responses after students submit their test booklets to school officials for scoring. That is currently an open question, but based on known previous examples in various parts of the United States, there is enough smoke here to prompt journalists to look for sources of fire.
- droithomme 15y ago
  Interesting, I didn't catch from the article that this three-year bump model had been tested on multiple incidents and found to correlate. It mentions in the third paragraph their methodology used "a pattern that, in Atlanta, indicated cheating in multiple schools". It seems Atlanta was the only place they are claiming there was a correlation, but even of this single data point they offer no specific evidence beyond the red dot in their graph. I did find it interesting that the biggest recent case where there was systematic corruption within a system was in Atlanta, and the newspaper here trying to show that cheating is common throughout the rest of the US and not just something special in Atlanta is Atlanta's leading newspaper. Recent analyses of data on 18,000 teachers in New Yock City has shown that year to year testing results are not as consistent as one would assume. http://news.firedoglake.com/2012/03/07/data-analysis-of-value-added-teacher-model-shows-no-correlation/ http://news.firedoglake.com/2012/03/07/data-analysis-of-valu... > Gary Rubinstein has been analyzing the data at his site, and he’s found some incredible results. First, he found that there’s almost no correlation among teachers year-over-year in the data. A teacher is likely to be judged as effective one year and ineffective the next. The average change in performance was a relatively large 25 points, and it did not fit with commonly accepted beliefs that teachers improve performance over time, particularly between the first and second year. http://garyrubinstein.teachforus.org/2012/02/26/analyzing-released-nyc-value-added-data-part-1/ http://garyrubinstein.teachforus.org/2012/02/26/analyzing-re... > Looking over the data, I found that 50% of the teachers had a 21 point ‘swing’ one way or the other. There were even teachers who had gone up or down as much as 80 points. The average change was 25 points. I also noticed that 49% of the teachers got lower value-added in 2010 than they did in 2009, contrary to my experience that most teachers improve from year to year. > With a correlation coefficient of .35, the scatter plot shows that teachers are not consistent from year to year ... nor do a good number of them go up These sorts of results where the data is available to the public for analysis makes me skeptical that testing results are actually consistent from year to year and that inconsistencies are evidence of cheating. I contrast these findings to the AJC article's claim that "Experts say student learning doesn’t typically jump backwards." The article does not state which experts they refer to, what their qualifications are, or what exactly they have said. That is suspicious to me, especially since the article has many quotes where they are no so shy about identifying their sources. Shouldn't it be even more critical to identify specific sources when using an appeal to authority argument citing "experts"? Especially when in cases where we have actual data, such as the study I cite above, scores did in fact jump backwards 49% of the time. This difference between empirical data and vague assertions by unnamed experts is suspicious. I must wonder who are these experts. Mind you, I think there is cheating. I am highly skeptical that the AJC's method has been shown to be a valid way to identify where it is happening.
glesica 15y ago
I didn't look too closely at this, but didn't you state that this was actually reported for schools, not districts? So then it could actually be that a single teacher could cause the "bump", especially in rural areas. I might be misinterpreting you though...

tokenadult 15y ago

I appreciate your thoughtful comments, droithomme, on many of the HN threads on education reform. You are correct that the statistical model used by the newspaper report needs more validation. On the other hand, as the story relates, the model is a generalization of a model developed during reporting by the same newspaper about known cheating incidents that could be independently verified by other means, including witness reports of overt conduct that breached test security or altered student-submitted answers after the test booklets were submitted to teachers or school principals. The earlier reporting by the same newspaper revealed that sometimes agreements ("conspiracies") to cheat on tests were rather widespread in entire schools or even entire school districts, and stayed secret for a time (at least) because all the school officials involved were united in having an interest in presenting fake test scores. I think that the scores bouncing up but then back down is, again, an empirical observation from the known cases of cheating in the area the newspaper reported on most closely a few years ago. As the submitted article here suggests, it is certainly notionally possible that a group of students might spurt forward suddenly in academic performance if they enjoy the instruction of a particularly good teacher. But, if genuine learning had taken place, it is notionally much harder to find the same group of students fall back the next year to a level lower than the level they reached the previous school year. It is much more likely that students might progress in grades from a teacher who is eager to look good, even if it means cheating, to a teacher who is not in on the cheating plot, or who at least is less effective at cheating in a manner that massively boosts scores. I think some of the other claims that you suggest would be made by the newspaper, if the statistical model were definitely backed up by other evidence of cheating, could be made if journalists in the various localities involved look for other sources of information. Not in all cases, as you correctly point out, but in many cases, additional reporting leg work could probably find direct evidence of violation of test security before the test resulting in exact "teaching to the test" item by item, or alteration of student item responses after students submit their test booklets to school officials for scoring. That is currently an open question, but based on known previous examples in various parts of the United States, there is enough smoke here to prompt journalists to look for sources of fire.

droithomme 15y ago
Interesting, I didn't catch from the article that this three-year bump model had been tested on multiple incidents and found to correlate. It mentions in the third paragraph their methodology used "a pattern that, in Atlanta, indicated cheating in multiple schools". It seems Atlanta was the only place they are claiming there was a correlation, but even of this single data point they offer no specific evidence beyond the red dot in their graph. I did find it interesting that the biggest recent case where there was systematic corruption within a system was in Atlanta, and the newspaper here trying to show that cheating is common throughout the rest of the US and not just something special in Atlanta is Atlanta's leading newspaper. Recent analyses of data on 18,000 teachers in New Yock City has shown that year to year testing results are not as consistent as one would assume. http://news.firedoglake.com/2012/03/07/data-analysis-of-value-added-teacher-model-shows-no-correlation/ http://news.firedoglake.com/2012/03/07/data-analysis-of-valu... > Gary Rubinstein has been analyzing the data at his site, and he’s found some incredible results. First, he found that there’s almost no correlation among teachers year-over-year in the data. A teacher is likely to be judged as effective one year and ineffective the next. The average change in performance was a relatively large 25 points, and it did not fit with commonly accepted beliefs that teachers improve performance over time, particularly between the first and second year. http://garyrubinstein.teachforus.org/2012/02/26/analyzing-released-nyc-value-added-data-part-1/ http://garyrubinstein.teachforus.org/2012/02/26/analyzing-re... > Looking over the data, I found that 50% of the teachers had a 21 point ‘swing’ one way or the other. There were even teachers who had gone up or down as much as 80 points. The average change was 25 points. I also noticed that 49% of the teachers got lower value-added in 2010 than they did in 2009, contrary to my experience that most teachers improve from year to year. > With a correlation coefficient of .35, the scatter plot shows that teachers are not consistent from year to year ... nor do a good number of them go up These sorts of results where the data is available to the public for analysis makes me skeptical that testing results are actually consistent from year to year and that inconsistencies are evidence of cheating. I contrast these findings to the AJC article's claim that "Experts say student learning doesn’t typically jump backwards." The article does not state which experts they refer to, what their qualifications are, or what exactly they have said. That is suspicious to me, especially since the article has many quotes where they are no so shy about identifying their sources. Shouldn't it be even more critical to identify specific sources when using an appeal to authority argument citing "experts"? Especially when in cases where we have actual data, such as the study I cite above, scores did in fact jump backwards 49% of the time. This difference between empirical data and vague assertions by unnamed experts is suspicious. I must wonder who are these experts. Mind you, I think there is cheating. I am highly skeptical that the AJC's method has been shown to be a valid way to identify where it is happening.

glesica 15y ago

I didn't look too closely at this, but didn't you state that this was actually reported for schools, not districts? So then it could actually be that a single teacher could cause the "bump", especially in rural areas. I might be misinterpreting you though...