5 ms·

So what's your recommended alternative metric? I don't doubt there is a good case to improve certain standardized tests, but that doesn't mean there isn't a val

So what's your recommended alternative metric? I don't doubt there is a good case to improve certain standardized tests, but that doesn't mean there isn't a valuable correlation even in a flawed one, nor that it's better to keep moving the goal posts into more subjective territory to explain away bad outcomes.

swivelmaster 9y ago
They're mentioned in the article. Graduation rates, college acceptance, stuff like that. Some of the countries with the best education systems in the world don't have these kinds of standardized tests. Also, nobody's suggesting moving the goal posts to explain away bad outcomes. If every school suddenly loses three points on average from one year to another on what is supposed to be the same test, the only common variable there is the test itself. The test HAS to be different from year to year in order not to repeat questions, but it HAS to remain a comparable difficulty.
- mulmen 9y ago
  Graduation rate strikes me as a dangerous metric because it could easily be maximized by lowering the bar. We want to graduate as many students as possible but we should not compromise quality to do so. It seems to me we need metrics on both. I do admit to not knowing what I am talking about so maybe there is no danger in using the graduation rate metric.
  - swivelmaster 9y ago
    Well the best metric is going to be something like annual pre-tax income a decade after graduation but that will never happen so we'll just need to figure out some kind of composite metric that avoids perverse incentives. edit: This is not an entirely serious suggestion. See later comment below.
    - mulmen 9y ago
      Should income really be the final metric? What about less tangible (measurable?) factors like happiness? If we program all of our students to maximize income do we really make the world a better place?
      - swivelmaster 9y ago
        It's not a metric for the students though, it's for the school. If you average it among all students, you get a reasonable metric for the how the school is doing. Of course, this would likely fail if you applied it to arts schools... Anyway, it wasn't a completely serious suggestion. It's got some problems. Happiness is a good thing to measure, but you'd want to still do some long term tracking of other metrics. Incarceration rates, income, college graduation rates, higher degrees and certifications, etc. edit: I make the comment about arts schools having received a degree in film and taken a haphazard path into startups and programming that ultimately worked out pretty well. I absolutely encourage people to go to school for the arts if that's what you want to be doing.
      - humanrebar 9y ago
        > What about less tangible (measurable?) factors like happiness? A primary goal of school is making kids smarter and ultimately a functioning adult. Including being employable. I don't think "happiness" is the job of an educator. The parents, absolutely. The local government? Maybe. But if "good citizen" is a desired goal, we might be able to measure that. With surveys of alumni about current events, employment rates, financial principles, etc. To measure whether well-equipped, life-long learners are produced. But I'd rather schools stay out of holistic goals better reserved for families.
      - rtkwe 9y ago
        Ignoring all that the timeline for that is so long it's not really useful for evaluating any changes because it'll be 10+ years before you have any new data, depending on where the change happens. On top of that it's so broad that it can only really measure the whole system from elementary up through college/trade-schools so even once you get data you can only really evaluate the system as a whole.
- lr4444lr 9y ago
  Graduation rates and college acceptance were tinkered with in New York to dismal results. Educational administrators were perversely incentivized, and it has showed up in atrociously high failure of local students to test out of remedial courses in public colleges. Just because the test mean drops does not mean the test is flawed. The educational inputs could simply have worsened, which is exactly what a reliable indicator should show.
- pkaye 9y ago
  Can you provide a reference to those countries with the best education systems?
  - tptacek 9y ago
    Finland has just one, an exit exam at the end of high school, like a mandatory SAT.
- secabeen 9y ago
  Do you have a metric that doesn't require waiting until students are leaving high school to return results? Some of these scores are for primary school students. It's not very useful to their teachers to be told 5-10 years after they teach a student that the improvements in instruction they attempted had or failed to have an effect.
dragonwriter 9y ago
The existing test is privately developed, opaque, and the responsible policy makers in the states using it, including California, can't get information on the validation from the maker. And, regardless of previous trends, which varied, every state using it saw a drop in 2017. This is all mentioned in the article. Either there was a common national effect that dropped real outcomes, all states using the test independently made simultaneously missteps that dropped real outcomes, or the 2017 test was changed in some way which produced the downward movement. If it weren't for the opacity, it would be easy to eliminate the last possibility.
criddell 9y ago
> So what's your recommended alternative metric? AFAIK, good test scores correlate well with college success, but don't say much about quality of life beyond that. There might not be a good metric. For a lot of people, one or two great teachers have more of an impact than anything that can be tested for.

swivelmaster 9y ago

They're mentioned in the article. Graduation rates, college acceptance, stuff like that. Some of the countries with the best education systems in the world don't have these kinds of standardized tests. Also, nobody's suggesting moving the goal posts to explain away bad outcomes. If every school suddenly loses three points on average from one year to another on what is supposed to be the same test, the only common variable there is the test itself. The test HAS to be different from year to year in order not to repeat questions, but it HAS to remain a comparable difficulty.

mulmen 9y ago
Graduation rate strikes me as a dangerous metric because it could easily be maximized by lowering the bar. We want to graduate as many students as possible but we should not compromise quality to do so. It seems to me we need metrics on both. I do admit to not knowing what I am talking about so maybe there is no danger in using the graduation rate metric.
- swivelmaster 9y ago
  Well the best metric is going to be something like annual pre-tax income a decade after graduation but that will never happen so we'll just need to figure out some kind of composite metric that avoids perverse incentives. edit: This is not an entirely serious suggestion. See later comment below.
  - mulmen 9y ago
    Should income really be the final metric? What about less tangible (measurable?) factors like happiness? If we program all of our students to maximize income do we really make the world a better place?
    - swivelmaster 9y ago
      It's not a metric for the students though, it's for the school. If you average it among all students, you get a reasonable metric for the how the school is doing. Of course, this would likely fail if you applied it to arts schools... Anyway, it wasn't a completely serious suggestion. It's got some problems. Happiness is a good thing to measure, but you'd want to still do some long term tracking of other metrics. Incarceration rates, income, college graduation rates, higher degrees and certifications, etc. edit: I make the comment about arts schools having received a degree in film and taken a haphazard path into startups and programming that ultimately worked out pretty well. I absolutely encourage people to go to school for the arts if that's what you want to be doing.
    - humanrebar 9y ago
      > What about less tangible (measurable?) factors like happiness? A primary goal of school is making kids smarter and ultimately a functioning adult. Including being employable. I don't think "happiness" is the job of an educator. The parents, absolutely. The local government? Maybe. But if "good citizen" is a desired goal, we might be able to measure that. With surveys of alumni about current events, employment rates, financial principles, etc. To measure whether well-equipped, life-long learners are produced. But I'd rather schools stay out of holistic goals better reserved for families.
    - rtkwe 9y ago
      Ignoring all that the timeline for that is so long it's not really useful for evaluating any changes because it'll be 10+ years before you have any new data, depending on where the change happens. On top of that it's so broad that it can only really measure the whole system from elementary up through college/trade-schools so even once you get data you can only really evaluate the system as a whole.
lr4444lr 9y ago
Graduation rates and college acceptance were tinkered with in New York to dismal results. Educational administrators were perversely incentivized, and it has showed up in atrociously high failure of local students to test out of remedial courses in public colleges. Just because the test mean drops does not mean the test is flawed. The educational inputs could simply have worsened, which is exactly what a reliable indicator should show.
pkaye 9y ago
Can you provide a reference to those countries with the best education systems?
- tptacek 9y ago
  Finland has just one, an exit exam at the end of high school, like a mandatory SAT.
secabeen 9y ago
Do you have a metric that doesn't require waiting until students are leaving high school to return results? Some of these scores are for primary school students. It's not very useful to their teachers to be told 5-10 years after they teach a student that the improvements in instruction they attempted had or failed to have an effect.

dragonwriter 9y ago

The existing test is privately developed, opaque, and the responsible policy makers in the states using it, including California, can't get information on the validation from the maker. And, regardless of previous trends, which varied, every state using it saw a drop in 2017. This is all mentioned in the article. Either there was a common national effect that dropped real outcomes, all states using the test independently made simultaneously missteps that dropped real outcomes, or the 2017 test was changed in some way which produced the downward movement. If it weren't for the opacity, it would be easy to eliminate the last possibility.

criddell 9y ago

> So what's your recommended alternative metric? AFAIK, good test scores correlate well with college success, but don't say much about quality of life beyond that. There might not be a good metric. For a lot of people, one or two great teachers have more of an impact than anything that can be tested for.