6 ms·

The problem is that the objective standards are harder to measure then you would think. How are you judging the urban teacher against the suburban one? Is the

The problem is that the objective standards are harder to measure then you would think. How are you judging the urban teacher against the suburban one? Is the kid who is getting abused at home ever going to match the one with two great parents? What percentage of your students eat breakfast? What percentage of your students have English as their first language? What percentage of the individual class are troublemakers? What tools are teachers allowed to use to subdue them? Principals differ on this. Are there bad teachers? Certainly! Do the metrics identify which teachers are the bad ones? Often, no.

yummyfajitas 16y ago
How are you judging the urban teacher against the suburban one? From my first post: For teacher..., they can use Value Added Modeling. Come up with a statistical predictor of performance, based on observable quantities such as income, ESL, race, free lunch, and previous year's performance. Teacher alpha = actual performance - predicted performance. All the factors you described are either highly correlated with easily available quantities (low income), or they are directly measurable. Thus, the predictor performance is likely to include them. I'd be curious to see if you could even hypothesize a systematic error (as opposed to sampling error) with a large effect that is not highly correlated with with easily observable base quantities (income, demographics, prior performance).
- jbooth 16y ago
  I do think that's the general direction things should go in.. But that statistical predictor of performance will always be horribly flawed. Applying a rigid set of a dozen statistics to human beings always misses the mark in a ton of cases. The question is whether it's better than the current system -- personally I'd want to see several iterations experimented with before I'd feel confident at all that it wasn't making things worse. "First, do no harm" and all that. One huge problem a lot of good urban teachers have with testing is that the tests as written really don't "speak to" poor urban and minority students. The teachers complain that they spend more time teaching the students how to take the test and less time teaching them the material in a way that's relevant to their lives. I haven't ever taught, so I'm just repeating this, but I can see where they're coming from.
  - yummyfajitas 16y ago
    Applying a rigid set of a dozen statistics to human beings always misses the mark in a ton of cases. http://en.wikipedia.org/wiki/Law_of_large_numbers http://en.wikipedia.org/wiki/Law_of_large_numbers The question is whether it's better than the current system... Without objective measurements, how could we ever answer this question?
    - jbooth 16y ago
      Teachers don't see that many kids in a year. Breaking them down into a combination of a few demographics is oversimplifying things to such a degree that I really don't get how anyone can be so blase about it. I mean, just to start with: * Are our tests measuring the right things? * Are we measuring the right things about the kids? * Are we capturing all of the data for the things we do measure about the kids? Those are huge unanswered questions to just be like "oh yeah just average the results for the income quintile, bang, done, we know who the good teachers are". If a student's being sexually abused at home, and test scores dramatically decline year-over-year, is the teacher a worse teacher for it? Just one edge case. Throw in 4 more and you've got a significant fraction of the scores that the teacher will be evaluated on. These are students' futures and teachers' jobs we're talking about, remember.. I think general teacher resistance to having some sort of oversimplified system shoved on them from above is a feature -- that system needs to be good enough to buy them in before being implemented.
      - yummyfajitas 16y ago
        Lets crunch some simple numbers. Suppose 1 in 100 children is sexually abused for the first time in any given semester [1], and a teacher has 4 classes of 30 students each. Then in 2.5% of classes, a teacher will have 4 sexually abused children (vs an average of 1.2). When this occurs, the percentage of children failing will increase by 2.3%. So lets say on average, a teacher is expected to have a failure rate of 20%. 2.5% of the time (roughly once in their career) they get unlucky and have a failure rate of 22.3% instead of 20%. This will occur 2 semesters in a row about one time in 1500, i.e. it will happen to 1 in 50 teachers (assuming a 30 year teaching career). Alternative numbers, to show that I'm not using cherry picked numbers: P(abuse)=0.001, P(4 abuses in class of 120) = 2.56e-6, slightly greater impact (failure rate goes up to 23.3% over 20%). If sexual abuse happens at a higher rate (e.g. 2%), then we get into the rather implausible situation that 48% of children are sexually abused, resulting in them failing school. [1] I don't know the relative proportions, but this seems like a high number. Among other things, it would imply that at least 24% of children are sexually abused, resulting in them failing school. We are also assuming the child has a good upbringing and good grades until the sexual abuse, which I imagine is not the most common situation. The assumption that it is the first instance of sexual abuse is important, because if it occurred in the prior year, it would likely affect their prior year grades and hence their current predicted grades.
- sethg 16y ago
  I can see this working to compare one school district to another or maybe even one large school to another, but I would balk at applying it to individual teachers. You can’t assume that after all the measurable factors in your statistical model are taken into account, the only remaining input to performance is the skill of the teacher. And even if the model was good at measuring teaching performance on average, that could just mean that out of every hundred teachers evaluated, two got higher alphas than they deserved and two got lower alphas.
  - yummyfajitas 16y ago
    P < 0.05? That's terrible! How will the education system survive? On the other hand, lets compare it to our current system of ignoring teacher quality. Assuming 25% of teachers are significantly above average and 25% are below (anyone have data on this?), that's a 50% error rate. Let me just address your main statistical misconception, however: You can’t assume that after all the measurable factors in your statistical model are taken into account, the only remaining input to performance is the skill of the teacher. This is not an assumption. The assumption is that after all the measurable factors are taken into account, the remainder are unbiased (i.e., have mean 0) or at least have bias smaller than ignoring all data. Can you hypothesize an external factor which would, in a single measurement period (either a semester or year), significantly reduce the scores of 20-30 students of a single teacher while not affecting the scores of other demographically similar students? And do you really believe this occurs so often that it would make measurement worse than ignoring all data?
    - sethg 16y ago
      “Either use Value Added Modeling or ignore all data” is a false dichotomy.
    - v21 16y ago
      > Can you hypothesize an external factor which would, in a single measurement period (either a semester or year), significantly reduce the scores of 20-30 students of a single teacher while not affecting the scores of other demographically similar students? Yeah. Classes, like any small social group of people, have their own dynamics. While the teacher should be able to exercise some control and restraint, they'll obviously be less successful as teachers if that's where their energy is going. Whereas a well-behaved class, they can focus all their energy on teaching. If you adjust for the socioeconomic factors of each student individually, you can separate them out, but you can't separate out the overall dynamics of the group from the performance of the teacher.

yummyfajitas 16y ago

How are you judging the urban teacher against the suburban one? From my first post: For teacher..., they can use Value Added Modeling. Come up with a statistical predictor of performance, based on observable quantities such as income, ESL, race, free lunch, and previous year's performance. Teacher alpha = actual performance - predicted performance. All the factors you described are either highly correlated with easily available quantities (low income), or they are directly measurable. Thus, the predictor performance is likely to include them. I'd be curious to see if you could even hypothesize a systematic error (as opposed to sampling error) with a large effect that is not highly correlated with with easily observable base quantities (income, demographics, prior performance).

jbooth 16y ago
I do think that's the general direction things should go in.. But that statistical predictor of performance will always be horribly flawed. Applying a rigid set of a dozen statistics to human beings always misses the mark in a ton of cases. The question is whether it's better than the current system -- personally I'd want to see several iterations experimented with before I'd feel confident at all that it wasn't making things worse. "First, do no harm" and all that. One huge problem a lot of good urban teachers have with testing is that the tests as written really don't "speak to" poor urban and minority students. The teachers complain that they spend more time teaching the students how to take the test and less time teaching them the material in a way that's relevant to their lives. I haven't ever taught, so I'm just repeating this, but I can see where they're coming from.
- yummyfajitas 16y ago
  Applying a rigid set of a dozen statistics to human beings always misses the mark in a ton of cases. http://en.wikipedia.org/wiki/Law_of_large_numbers http://en.wikipedia.org/wiki/Law_of_large_numbers The question is whether it's better than the current system... Without objective measurements, how could we ever answer this question?
  - jbooth 16y ago
    Teachers don't see that many kids in a year. Breaking them down into a combination of a few demographics is oversimplifying things to such a degree that I really don't get how anyone can be so blase about it. I mean, just to start with: * Are our tests measuring the right things? * Are we measuring the right things about the kids? * Are we capturing all of the data for the things we do measure about the kids? Those are huge unanswered questions to just be like "oh yeah just average the results for the income quintile, bang, done, we know who the good teachers are". If a student's being sexually abused at home, and test scores dramatically decline year-over-year, is the teacher a worse teacher for it? Just one edge case. Throw in 4 more and you've got a significant fraction of the scores that the teacher will be evaluated on. These are students' futures and teachers' jobs we're talking about, remember.. I think general teacher resistance to having some sort of oversimplified system shoved on them from above is a feature -- that system needs to be good enough to buy them in before being implemented.
    - yummyfajitas 16y ago
      Lets crunch some simple numbers. Suppose 1 in 100 children is sexually abused for the first time in any given semester [1], and a teacher has 4 classes of 30 students each. Then in 2.5% of classes, a teacher will have 4 sexually abused children (vs an average of 1.2). When this occurs, the percentage of children failing will increase by 2.3%. So lets say on average, a teacher is expected to have a failure rate of 20%. 2.5% of the time (roughly once in their career) they get unlucky and have a failure rate of 22.3% instead of 20%. This will occur 2 semesters in a row about one time in 1500, i.e. it will happen to 1 in 50 teachers (assuming a 30 year teaching career). Alternative numbers, to show that I'm not using cherry picked numbers: P(abuse)=0.001, P(4 abuses in class of 120) = 2.56e-6, slightly greater impact (failure rate goes up to 23.3% over 20%). If sexual abuse happens at a higher rate (e.g. 2%), then we get into the rather implausible situation that 48% of children are sexually abused, resulting in them failing school. [1] I don't know the relative proportions, but this seems like a high number. Among other things, it would imply that at least 24% of children are sexually abused, resulting in them failing school. We are also assuming the child has a good upbringing and good grades until the sexual abuse, which I imagine is not the most common situation. The assumption that it is the first instance of sexual abuse is important, because if it occurred in the prior year, it would likely affect their prior year grades and hence their current predicted grades.
sethg 16y ago
I can see this working to compare one school district to another or maybe even one large school to another, but I would balk at applying it to individual teachers. You can’t assume that after all the measurable factors in your statistical model are taken into account, the only remaining input to performance is the skill of the teacher. And even if the model was good at measuring teaching performance on average, that could just mean that out of every hundred teachers evaluated, two got higher alphas than they deserved and two got lower alphas.
- yummyfajitas 16y ago
  P < 0.05? That's terrible! How will the education system survive? On the other hand, lets compare it to our current system of ignoring teacher quality. Assuming 25% of teachers are significantly above average and 25% are below (anyone have data on this?), that's a 50% error rate. Let me just address your main statistical misconception, however: You can’t assume that after all the measurable factors in your statistical model are taken into account, the only remaining input to performance is the skill of the teacher. This is not an assumption. The assumption is that after all the measurable factors are taken into account, the remainder are unbiased (i.e., have mean 0) or at least have bias smaller than ignoring all data. Can you hypothesize an external factor which would, in a single measurement period (either a semester or year), significantly reduce the scores of 20-30 students of a single teacher while not affecting the scores of other demographically similar students? And do you really believe this occurs so often that it would make measurement worse than ignoring all data?
  - sethg 16y ago
    “Either use Value Added Modeling or ignore all data” is a false dichotomy.
  - v21 16y ago
    > Can you hypothesize an external factor which would, in a single measurement period (either a semester or year), significantly reduce the scores of 20-30 students of a single teacher while not affecting the scores of other demographically similar students? Yeah. Classes, like any small social group of people, have their own dynamics. While the teacher should be able to exercise some control and restraint, they'll obviously be less successful as teachers if that's where their energy is going. Whereas a well-behaved class, they can focus all their energy on teaching. If you adjust for the socioeconomic factors of each student individually, you can separate them out, but you can't separate out the overall dynamics of the group from the performance of the teacher.