5 ms·

So replace a metric with 0.1 correlation with one having 0.0 correlation? Two problems with this: 1. The other metric almost certainly doesn't have an 0.0 cor

So replace a metric with 0.1 correlation with one having 0.0 correlation? Two problems with this: 1. The other metric almost certainly doesn't have an 0.0 correlation, but we don't know that for sure since the data wasn't released for whatever reason. 2. The metric with 0.1 correlation (or whatever the number is)... keep in mind the context. What is the correlation with? Test scores, something that can be and is often rigged, and is at best tangentially related to the ultimate goal which is inherently qualitative in nature. I think the comparison to programming is instructive. If you had 1/3 of your salary determined by the LOC you wrote or the number of bugs you closed, you would game the system and maximize your salary, even if you weren't providing value by doing that and even if those naive metrics did very weakly correlate with performance in some big study prior to implementing the monetary incentive. What would be gained in that scenario? If you are advocating that we should use the school principal's opinion rather than VAM, why do you believe opinion is superior? Do you have evidence that principal's opinion has a higher correlation with student outcomes than VAM? One of the points made in the blog I linked to is that it's unfortunate that they didn't release that data. And as I said above, it's not straightforward to assume that these weak correlations are actually meaningful in practice.

yummyfajitas 14y ago
The 0.3 correlation described in the article is the correlation between a teacher's VAM score in a single class last year and this year. The author is arguing that because the measurement is noisy, we should ignore it. This is silly - it just means a single class-year's point estimate is noisy, and multiple class-years must be combined to form an accurate estimate. If you had 1/3 of your salary determined by the LOC you wrote or the number of bugs you closed, you would game the system and maximize your salary... Indeed - if 1/3 of my salary was determined by the number of kids in my class who can read better than when they are predicted to read at this age, I'd definitely try to make sure that reading skills improved. Conversely, if a stupid metric such as student/principal opinion were used, I'd focus on jokes and friendliness over education. (Well, actually I didn't back when I taught. But that sure didn't help my student evaluations...)
- streptomycin 14y ago
  The author is arguing that because the measurement is noisy, we should ignore it. Not quite.. Indeed - if 1/3 of my salary was determined by the number of kids in my class who can read better than when they are predicted to read at this age, I'd definitely try to make sure that reading skills improved. Keep in mind that "score on a reading test" and "reading skills" are not the same things. For instance, one would be significantly improved by spending valuable class time teaching students tricks about how to succeed on the standardized reading test. It's similar to the incentives you'd get by determining salary by LOC - even if LOC correlates with performance, basing pay on it encourages all kinds of nonsense that doesn't actually benefit anyone. So in that context, we have a metric that very loosely correlates with this clearly flawed marker of success. And we're supposed to spend ungodly sums of money implementing this strategy? Come on. Conversely, if a stupid metric such as student/principal opinion were used, I'd focus on jokes and friendliness over education. Well, they do propose to use student and principal evaluations..
  - yummyfajitas 14y ago
    Keep in mind that "score on a reading test" and "reading skills" are not the same things. True. Score on a reading test might be lower than actual reading skills due to a lack of test taking skills. Then again, I can't think of any measurable quantity more highly correlated with reading skills than scoring high on a reading test. Do you have evidence that "principal's opinion of teacher" is better? ...we have a metric that very loosely correlates... You are confused. GR shows that year on year correlations are weak (0.3), not that correlation between reading test and reading skills are weak. You are correct that a certain nonzero amount of teaching test skills will be effective at improving scores. I see no reason to believe this amount will be large - do you? Again, standardized tests are not the only gameable metric. Principal/student opinion is as well. Why do you believe they are less gameable and more accurate than directly measuring student ability?
    - streptomycin 14y ago
      It's not that I think this is the only flawed metric or that there are other methods that are quantifiably better. It's that we've spent ridiculous sums of time and money on this complicated new metric, despite some pretty clear arguments against it, and we still are very far away from actually proving that it is better than the status quo, even if that status quo obviously falls somewhat short of perfection. It's like.. how much money and effort do you think is worthwhile to spend on something with such uncertain prospects? There are tons of great grant proposals to funding agencies which are rejected. Maybe we would be better off funding some of those. Opportunity costs exist, and it seems like undue sums of money are being wasted on this stuff.

yummyfajitas 14y ago

The 0.3 correlation described in the article is the correlation between a teacher's VAM score in a single class last year and this year. The author is arguing that because the measurement is noisy, we should ignore it. This is silly - it just means a single class-year's point estimate is noisy, and multiple class-years must be combined to form an accurate estimate. If you had 1/3 of your salary determined by the LOC you wrote or the number of bugs you closed, you would game the system and maximize your salary... Indeed - if 1/3 of my salary was determined by the number of kids in my class who can read better than when they are predicted to read at this age, I'd definitely try to make sure that reading skills improved. Conversely, if a stupid metric such as student/principal opinion were used, I'd focus on jokes and friendliness over education. (Well, actually I didn't back when I taught. But that sure didn't help my student evaluations...)

streptomycin 14y ago
The author is arguing that because the measurement is noisy, we should ignore it. Not quite.. Indeed - if 1/3 of my salary was determined by the number of kids in my class who can read better than when they are predicted to read at this age, I'd definitely try to make sure that reading skills improved. Keep in mind that "score on a reading test" and "reading skills" are not the same things. For instance, one would be significantly improved by spending valuable class time teaching students tricks about how to succeed on the standardized reading test. It's similar to the incentives you'd get by determining salary by LOC - even if LOC correlates with performance, basing pay on it encourages all kinds of nonsense that doesn't actually benefit anyone. So in that context, we have a metric that very loosely correlates with this clearly flawed marker of success. And we're supposed to spend ungodly sums of money implementing this strategy? Come on. Conversely, if a stupid metric such as student/principal opinion were used, I'd focus on jokes and friendliness over education. Well, they do propose to use student and principal evaluations..
- yummyfajitas 14y ago
  Keep in mind that "score on a reading test" and "reading skills" are not the same things. True. Score on a reading test might be lower than actual reading skills due to a lack of test taking skills. Then again, I can't think of any measurable quantity more highly correlated with reading skills than scoring high on a reading test. Do you have evidence that "principal's opinion of teacher" is better? ...we have a metric that very loosely correlates... You are confused. GR shows that year on year correlations are weak (0.3), not that correlation between reading test and reading skills are weak. You are correct that a certain nonzero amount of teaching test skills will be effective at improving scores. I see no reason to believe this amount will be large - do you? Again, standardized tests are not the only gameable metric. Principal/student opinion is as well. Why do you believe they are less gameable and more accurate than directly measuring student ability?
  - streptomycin 14y ago
    It's not that I think this is the only flawed metric or that there are other methods that are quantifiably better. It's that we've spent ridiculous sums of time and money on this complicated new metric, despite some pretty clear arguments against it, and we still are very far away from actually proving that it is better than the status quo, even if that status quo obviously falls somewhat short of perfection. It's like.. how much money and effort do you think is worthwhile to spend on something with such uncertain prospects? There are tons of great grant proposals to funding agencies which are rejected. Maybe we would be better off funding some of those. Opportunity costs exist, and it seems like undue sums of money are being wasted on this stuff.