3 ms·

> Except that they already are measuring achievement They are measuring performance on standardized tests. If by "achievement" you mean "does well on multiple

> Except that they already are measuring achievement They are measuring performance on standardized tests. If by "achievement" you mean "does well on multiple choice exams", then sure. I don't trust standarized tests as a metric of achievement. Even without metric gaming they are pretty awful. And we've been solidly in systematic metric gaming land for least the past two decades. The entire K12 system is designed around increasing these test scores. > If you count expenditure as a positive, you assume your conclusion that spending is good. No, you assume there's a hidden function that you can't measure or explain but which is representative of reality and also positively correlates funding levels with performance. I know it's a radical proposal in STEM communities, but sometimes measuring things causes a lot more problems than it solves.

darawk 8y ago
> Do you think those are good measurement of "achievement"? Sometimes yes, sometimes no. But spending is never a good measurement of achievement. > No. My assumption is that measuring achievement is inherently impossible in something as multi-faceted and large as universal education. You think it is impossible to measure educational performance at all? Surely you don't believe something that silly. You think it is measured imperfectly. And I agree with you. But i'm not sure what your point is here. Certainly imperfect measurements don't mean we should just measure random, unrelated variables like spending. I also don't see how it means we shouldn't try to measure, and improve our measurement standards. And it certainly doesn't mean, that the conclusion of this article is in any way incorrect (which is not that the measurements are good, but that the way other people were using these measurements were even less good than using them correctly).
- thrmsforbfast 8y ago
  > But spending is never a good measurement of achievement. Nobody said it was. > You think it is impossible to measure educational performance at all? Surely you don't believe something that silly. Obviously not. That's a ridiculous strawman and I think you know it. > But i'm not sure what your point is here. My point is that the availability of resources IS a useful metric. > Certainly imperfect measurements don't mean we should just measure random, unrelated variables like spending. How is availability of resources unrelated? The idea that the level of resources available is irrelevant strikes me as fairly insane. You're really saying that you'd have the same quality teachers as $40k/yr as at $90k/yr or $500,000k/yr? CLEARLY resources effect outcomes. It's not the whole story, but it is a useful feature. To claim it's completely irrelevant because "oh look multiple choice exams that everyone has been systematically gaming for 20+ years" seems absolutely insane to me. > I also don't see how it means we shouldn't try to measure, and improve our measurement standards. Sure. > And it certainly doesn't mean, that the conclusion of this article is in any way incorrect (which is not that the measurements are good, but that the way other people were using these measurements were even less good than using them correctly). I think the fundamental thesis of the article -- that we can scientifically manage a system of universal education -- is fundamentally and impossibly flawed. Especially given the current state of assessments and the incentive structures built around them.
  - darawk 8y ago
    Oh, but they did, from the article: > We fixed two serious problems common to traditional rankings. First, we removed factors that do not measure K–12 student performance or teaching effectiveness, such as spending per student (intentions to raise performance are not the same as raising performance)
    - thrmsforbfast 8y ago
      If you want to build a school system around maximizing scores on a very particular type of standardized test then their methodology makes sense. Otherwise, it's just weighing one questionable feature more than another questionable feature. Test scores do NOT provide a holistic picture of the quality of a school system. Neither do funding levels. BOTH of those things can be increased while not improving or even harming actual educational achievement. They are FEATURES, and reasonable models can weigh them in different ways. The assertion that funding levels might be positively correlated with (unmeasurable -- you've conceded this point!) true outcomes is not unreasonable. The assessments the authors are criticizing actually have it completely right. Both test scores and funding levels are PROXIES for the actual quality of education that's happening. They are features. It's unclear how to weight these features against one another, but the conjecture that either of those features might be correlative or even causative is totally reasonable. The authors, however, go off the deep end and completely mistake one particular map (and a damn bad one at that) for the territory. If the authors' assertion made any sense, then we could replace all teachers with DRL.
      - darawk 8y ago
        > The assertion that funding levels might be positively correlated with (unmeasurable -- you've conceded this point!) true outcomes is not unreasonable. No it's not unreasonable, as a hypothesis. But if you wanted to, you know, test that hypothesis, guess what you'd need: performance measurement. Which means that the only way you can prove that spending matters is by measuring its impact on performance, which is already being measured by the other variables. So, there is zero reason to believe that incorporating spending as an additional variable can improve your assessment of quality.

darawk 8y ago

> Do you think those are good measurement of "achievement"? Sometimes yes, sometimes no. But spending is never a good measurement of achievement. > No. My assumption is that measuring achievement is inherently impossible in something as multi-faceted and large as universal education. You think it is impossible to measure educational performance at all? Surely you don't believe something that silly. You think it is measured imperfectly. And I agree with you. But i'm not sure what your point is here. Certainly imperfect measurements don't mean we should just measure random, unrelated variables like spending. I also don't see how it means we shouldn't try to measure, and improve our measurement standards. And it certainly doesn't mean, that the conclusion of this article is in any way incorrect (which is not that the measurements are good, but that the way other people were using these measurements were even less good than using them correctly).

thrmsforbfast 8y ago
> But spending is never a good measurement of achievement. Nobody said it was. > You think it is impossible to measure educational performance at all? Surely you don't believe something that silly. Obviously not. That's a ridiculous strawman and I think you know it. > But i'm not sure what your point is here. My point is that the availability of resources IS a useful metric. > Certainly imperfect measurements don't mean we should just measure random, unrelated variables like spending. How is availability of resources unrelated? The idea that the level of resources available is irrelevant strikes me as fairly insane. You're really saying that you'd have the same quality teachers as $40k/yr as at $90k/yr or $500,000k/yr? CLEARLY resources effect outcomes. It's not the whole story, but it is a useful feature. To claim it's completely irrelevant because "oh look multiple choice exams that everyone has been systematically gaming for 20+ years" seems absolutely insane to me. > I also don't see how it means we shouldn't try to measure, and improve our measurement standards. Sure. > And it certainly doesn't mean, that the conclusion of this article is in any way incorrect (which is not that the measurements are good, but that the way other people were using these measurements were even less good than using them correctly). I think the fundamental thesis of the article -- that we can scientifically manage a system of universal education -- is fundamentally and impossibly flawed. Especially given the current state of assessments and the incentive structures built around them.
- darawk 8y ago
  Oh, but they did, from the article: > We fixed two serious problems common to traditional rankings. First, we removed factors that do not measure K–12 student performance or teaching effectiveness, such as spending per student (intentions to raise performance are not the same as raising performance)
  - thrmsforbfast 8y ago
    If you want to build a school system around maximizing scores on a very particular type of standardized test then their methodology makes sense. Otherwise, it's just weighing one questionable feature more than another questionable feature. Test scores do NOT provide a holistic picture of the quality of a school system. Neither do funding levels. BOTH of those things can be increased while not improving or even harming actual educational achievement. They are FEATURES, and reasonable models can weigh them in different ways. The assertion that funding levels might be positively correlated with (unmeasurable -- you've conceded this point!) true outcomes is not unreasonable. The assessments the authors are criticizing actually have it completely right. Both test scores and funding levels are PROXIES for the actual quality of education that's happening. They are features. It's unclear how to weight these features against one another, but the conjecture that either of those features might be correlative or even causative is totally reasonable. The authors, however, go off the deep end and completely mistake one particular map (and a damn bad one at that) for the territory. If the authors' assertion made any sense, then we could replace all teachers with DRL.
    - darawk 8y ago
      > The assertion that funding levels might be positively correlated with (unmeasurable -- you've conceded this point!) true outcomes is not unreasonable. No it's not unreasonable, as a hypothesis. But if you wanted to, you know, test that hypothesis, guess what you'd need: performance measurement. Which means that the only way you can prove that spending matters is by measuring its impact on performance, which is already being measured by the other variables. So, there is zero reason to believe that incorporating spending as an additional variable can improve your assessment of quality.