4 ms·

I wonder what limitations there are to using standardized testing for evaluating "progress"? From personal experience, many of my friends who score very well on

by andreyf 17y ago

I wonder what limitations there are to using standardized testing for evaluating "progress"? From personal experience, many of my friends who score very well on standardized tests aren't necessarily "smart" in the sense Feynman was "smart" - that is, capable of creating and sharpening their models of the world. What's ironic is that while Teach for America acknowledges that it's extremely difficult to measure what makes a "good teacher", in that same sense of a person who continuously creates and sharpens their models of the world, they seem to assume it's trivial to measure a "good student", even though the two are fundamentally the same.

patio11 17y ago
I wonder what limitations there are to using standardized testing for evaluating "progress"? Any heuristic you care to use is significantly better than the traditional metric for school quality, which is spending per pupil. (This metric was selected because optimizing for it optimizes for the outcomes teachers unions desire. American public school exist to employ union teachers and sometimes cause education as an industrial biproduct.) This puts me in mind of the metric my day job used to track productivity for several decades: hours worked by the engineering staff. That one had perverse consequences, too. (Having a bad quarter? Need a good statistic to report to the head office? Start a deathmarch!)
- wisty 17y ago
  Goodheart's Law (not to be confused with Godwin or Murphy's Laws): "any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes". Randall Munroe (xkcd) has make a similar observation.
  - RyanMcGreal 17y ago
    As soon as you start measuring something and attaching organizational outcomes to the results, you encourage gaming.
    - tokenadult 17y ago
      As above, better that what people are pursuing is student learning than spending per school.
wisty 17y ago
It's not a teachers job to teach "smart". It's a teachers job to teach the subject, which gives the students more tools to make themselves "smart". I'm not sure if there is any way you could design a schooling system that teaches "smart", short of outlawing homework so students have more time to teach it to themselves. (I'm only half joking). There are a number of levels in teaching (and learning). You learn the basic facts (or processes), you learn to apply them in simple situations, and you learn to apply them creatively. This can be tested by multiple-choice tests, or more in-depth tests, but multiple-choice is fairly reliable and very cheap. Besides, the difference between a "good" and "bad" student is pretty easy to measure, if you look at the broad population.
- andreyf 17y ago
  It's not a teachers job to teach "smart". Yes it is, by definition. Once people are "smart", in the Feynman sense, there's no need for anyone to teach them a subject - they're self-motivated learners that will seek them out on their own. You learn the basic facts (or processes), you learn to apply them in simple situations, and you learn to apply them creatively. Although somehow intuitive, I don't think that's a good model. What are "basic facts"? What is a "simple situation"? What does it mean to "apply facts creatively"? All of these are vaguely intuitive but not well defined, with no epistemological basis. There is a much better definition of learning - the ability to create models of our experiences, test, and refine them. I know of no reason a cheap test can't be devised to test such learning, nor of any attempts to create one.
  - wisty 17y ago
    We seem to be at loggerheads on two points. 1 - I'm focusing on basic proficiency (a bottom up approach - you learn the basics then the hard stuff is easy), you are focusing on learning through break-through discoveries (the top-down approach). (I may be creating a false hierarchy, but I'm going to run with it). 2 - I am focusing on teachers directly teaching students, while you focus on teaching students to be self-motivated learners. Now, I can't think of any way to teach self-motivated break-through learning. The closest thing I can think of is to make subjects interesting (getting teachers who are enthusiastic about the subject, and providing interesting text books (yeah-right)), and give the students free time to pursue their own interests (which is why I half-jokingly suggested outlawing homework). The other model I've seen (don't teach properly so students have to learn) seems a little perverse to me. (oh, the basic facts -> simple application -> creative use? Fact - past tense. Simple application - what did Spot do? Creative (or complex) use - Why did Spot do it?)
thrdOriginal 17y ago
The article definitely glossed over how they measure "good students" and this is a very real problem. When student success is measured only or mainly by standardized tests (and teacher and school success extrapolated from that) surprise surprise: teachers start teaching only to the test. Now you have students that are great at taking tests but have not been encouraged to develop those "Feynman smarts."
- tokenadult 17y ago
  Now you have students that are great at taking tests but have not been encouraged to develop those "Feynman smarts." Proof that one goal necessarily excludes the other? Right now we are getting neither, in most cases in most schools, so what Teach for America is looking at seems to be progress to me.
  - thrdOriginal 17y ago
    Sure these things don't, by definition, implicitly exclude each other, but in practice they do. How much time do you think teacher's have in a day? For proof talk to any good teacher, or sit in on a school board meeting.

by andreyf 17y ago

patio11 17y ago

I wonder what limitations there are to using standardized testing for evaluating "progress"? Any heuristic you care to use is significantly better than the traditional metric for school quality, which is spending per pupil. (This metric was selected because optimizing for it optimizes for the outcomes teachers unions desire. American public school exist to employ union teachers and sometimes cause education as an industrial biproduct.) This puts me in mind of the metric my day job used to track productivity for several decades: hours worked by the engineering staff. That one had perverse consequences, too. (Having a bad quarter? Need a good statistic to report to the head office? Start a deathmarch!)

wisty 17y ago
Goodheart's Law (not to be confused with Godwin or Murphy's Laws): "any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes". Randall Munroe (xkcd) has make a similar observation.
- RyanMcGreal 17y ago
  As soon as you start measuring something and attaching organizational outcomes to the results, you encourage gaming.
  - tokenadult 17y ago
    As above, better that what people are pursuing is student learning than spending per school.

wisty 17y ago

It's not a teachers job to teach "smart". It's a teachers job to teach the subject, which gives the students more tools to make themselves "smart". I'm not sure if there is any way you could design a schooling system that teaches "smart", short of outlawing homework so students have more time to teach it to themselves. (I'm only half joking). There are a number of levels in teaching (and learning). You learn the basic facts (or processes), you learn to apply them in simple situations, and you learn to apply them creatively. This can be tested by multiple-choice tests, or more in-depth tests, but multiple-choice is fairly reliable and very cheap. Besides, the difference between a "good" and "bad" student is pretty easy to measure, if you look at the broad population.

andreyf 17y ago
It's not a teachers job to teach "smart". Yes it is, by definition. Once people are "smart", in the Feynman sense, there's no need for anyone to teach them a subject - they're self-motivated learners that will seek them out on their own. You learn the basic facts (or processes), you learn to apply them in simple situations, and you learn to apply them creatively. Although somehow intuitive, I don't think that's a good model. What are "basic facts"? What is a "simple situation"? What does it mean to "apply facts creatively"? All of these are vaguely intuitive but not well defined, with no epistemological basis. There is a much better definition of learning - the ability to create models of our experiences, test, and refine them. I know of no reason a cheap test can't be devised to test such learning, nor of any attempts to create one.
- wisty 17y ago
  We seem to be at loggerheads on two points. 1 - I'm focusing on basic proficiency (a bottom up approach - you learn the basics then the hard stuff is easy), you are focusing on learning through break-through discoveries (the top-down approach). (I may be creating a false hierarchy, but I'm going to run with it). 2 - I am focusing on teachers directly teaching students, while you focus on teaching students to be self-motivated learners. Now, I can't think of any way to teach self-motivated break-through learning. The closest thing I can think of is to make subjects interesting (getting teachers who are enthusiastic about the subject, and providing interesting text books (yeah-right)), and give the students free time to pursue their own interests (which is why I half-jokingly suggested outlawing homework). The other model I've seen (don't teach properly so students have to learn) seems a little perverse to me. (oh, the basic facts -> simple application -> creative use? Fact - past tense. Simple application - what did Spot do? Creative (or complex) use - Why did Spot do it?)

thrdOriginal 17y ago

The article definitely glossed over how they measure "good students" and this is a very real problem. When student success is measured only or mainly by standardized tests (and teacher and school success extrapolated from that) surprise surprise: teachers start teaching only to the test. Now you have students that are great at taking tests but have not been encouraged to develop those "Feynman smarts."

tokenadult 17y ago
Now you have students that are great at taking tests but have not been encouraged to develop those "Feynman smarts." Proof that one goal necessarily excludes the other? Right now we are getting neither, in most cases in most schools, so what Teach for America is looking at seems to be progress to me.
- thrdOriginal 17y ago
  Sure these things don't, by definition, implicitly exclude each other, but in practice they do. How much time do you think teacher's have in a day? For proof talk to any good teacher, or sit in on a school board meeting.