3 ms·
I appreciate your thoughtful comments, droithomme, on many of the HN threads on education reform. You are correct that the statistical model used by the newspap
by tokenadult 15y ago
I appreciate your thoughtful comments, droithomme, on many of the HN threads on education reform. You are correct that the statistical model used by the newspaper report needs more validation. On the other hand, as the story relates, the model is a generalization of a model developed during reporting by the same newspaper about known cheating incidents that could be independently verified by other means, including witness reports of overt conduct that breached test security or altered student-submitted answers after the test booklets were submitted to teachers or school principals. The earlier reporting by the same newspaper revealed that sometimes agreements ("conspiracies") to cheat on tests were rather widespread in entire schools or even entire school districts, and stayed secret for a time (at least) because all the school officials involved were united in having an interest in presenting fake test scores.
I think that the scores bouncing up but then back down is, again, an empirical observation from the known cases of cheating in the area the newspaper reported on most closely a few years ago. As the submitted article here suggests, it is certainly notionally possible that a group of students might spurt forward suddenly in academic performance if they enjoy the instruction of a particularly good teacher. But, if genuine learning had taken place, it is notionally much harder to find the same group of students fall back the next year to a level lower than the level they reached the previous school year. It is much more likely that students might progress in grades from a teacher who is eager to look good, even if it means cheating, to a teacher who is not in on the cheating plot, or who at least is less effective at cheating in a manner that massively boosts scores.
I think some of the other claims that you suggest would be made by the newspaper, if the statistical model were definitely backed up by other evidence of cheating, could be made if journalists in the various localities involved look for other sources of information. Not in all cases, as you correctly point out, but in many cases, additional reporting leg work could probably find direct evidence of violation of test security before the test resulting in exact "teaching to the test" item by item, or alteration of student item responses after students submit their test booklets to school officials for scoring. That is currently an open question, but based on known previous examples in various parts of the United States, there is enough smoke here to prompt journalists to look for sources of fire.
- droithomme 15y agoInteresting, I didn't catch from the article that this three-year bump model had been tested on multiple incidents and found to correlate. It mentions in the third paragraph their methodology used "a pattern that, in Atlanta, indicated cheating in multiple schools". It seems Atlanta was the only place they are claiming there was a correlation, but even of this single data point they offer no specific evidence beyond the red dot in their graph. I did find it interesting that the biggest recent case where there was systematic corruption within a system was in Atlanta, and the newspaper here trying to show that cheating is common throughout the rest of the US and not just something special in Atlanta is Atlanta's leading newspaper. Recent analyses of data on 18,000 teachers in New Yock City has shown that year to year testing results are not as consistent as one would assume. http://news.firedoglake.com/2012/03/07/data-analysis-of-value-added-teacher-model-shows-no-correlation/ http://news.firedoglake.com/2012/03/07/data-analysis-of-valu... > Gary Rubinstein has been analyzing the data at his site, and he’s found some incredible results. First, he found that there’s almost no correlation among teachers year-over-year in the data. A teacher is likely to be judged as effective one year and ineffective the next. The average change in performance was a relatively large 25 points, and it did not fit with commonly accepted beliefs that teachers improve performance over time, particularly between the first and second year. http://garyrubinstein.teachforus.org/2012/02/26/analyzing-released-nyc-value-added-data-part-1/ http://garyrubinstein.teachforus.org/2012/02/26/analyzing-re... > Looking over the data, I found that 50% of the teachers had a 21 point ‘swing’ one way or the other. There were even teachers who had gone up or down as much as 80 points. The average change was 25 points. I also noticed that 49% of the teachers got lower value-added in 2010 than they did in 2009, contrary to my experience that most teachers improve from year to year. > With a correlation coefficient of .35, the scatter plot shows that teachers are not consistent from year to year ... nor do a good number of them go up These sorts of results where the data is available to the public for analysis makes me skeptical that testing results are actually consistent from year to year and that inconsistencies are evidence of cheating. I contrast these findings to the AJC article's claim that "Experts say student learning doesn’t typically jump backwards." The article does not state which experts they refer to, what their qualifications are, or what exactly they have said. That is suspicious to me, especially since the article has many quotes where they are no so shy about identifying their sources. Shouldn't it be even more critical to identify specific sources when using an appeal to authority argument citing "experts"? Especially when in cases where we have actual data, such as the study I cite above, scores did in fact jump backwards 49% of the time. This difference between empirical data and vague assertions by unnamed experts is suspicious. I must wonder who are these experts. Mind you, I think there is cheating. I am highly skeptical that the AJC's method has been shown to be a valid way to identify where it is happening.