3 ms·

I can see this working to compare one school district to another or maybe even one large school to another, but I would balk at applying it to individual teache

by sethg 16y ago

I can see this working to compare one school district to another or maybe even one large school to another, but I would balk at applying it to individual teachers. You can’t assume that after all the measurable factors in your statistical model are taken into account, the only remaining input to performance is the skill of the teacher. And even if the model was good at measuring teaching performance on average, that could just mean that out of every hundred teachers evaluated, two got higher alphas than they deserved and two got lower alphas.

yummyfajitas 16y ago
P < 0.05? That's terrible! How will the education system survive? On the other hand, lets compare it to our current system of ignoring teacher quality. Assuming 25% of teachers are significantly above average and 25% are below (anyone have data on this?), that's a 50% error rate. Let me just address your main statistical misconception, however: You can’t assume that after all the measurable factors in your statistical model are taken into account, the only remaining input to performance is the skill of the teacher. This is not an assumption. The assumption is that after all the measurable factors are taken into account, the remainder are unbiased (i.e., have mean 0) or at least have bias smaller than ignoring all data. Can you hypothesize an external factor which would, in a single measurement period (either a semester or year), significantly reduce the scores of 20-30 students of a single teacher while not affecting the scores of other demographically similar students? And do you really believe this occurs so often that it would make measurement worse than ignoring all data?
- sethg 16y ago
  “Either use Value Added Modeling or ignore all data” is a false dichotomy.
- v21 16y ago
  > Can you hypothesize an external factor which would, in a single measurement period (either a semester or year), significantly reduce the scores of 20-30 students of a single teacher while not affecting the scores of other demographically similar students? Yeah. Classes, like any small social group of people, have their own dynamics. While the teacher should be able to exercise some control and restraint, they'll obviously be less successful as teachers if that's where their energy is going. Whereas a well-behaved class, they can focus all their energy on teaching. If you adjust for the socioeconomic factors of each student individually, you can separate them out, but you can't separate out the overall dynamics of the group from the performance of the teacher.
  - yummyfajitas 16y ago
    If you adjust for the socioeconomic factors of each student individually, you can separate them out... This is precisely what VAM does. Each student gets a predicted grade and they are combined to get an overall class grade. If things like well-behavedness are correlated with socioeconomic status, they will be incorporated into the predictor. Thus, a low SES class will have a lower predicted grade, and their poor behavior will be part of the reason why. The only component of student behavior the predictor will not account for is whatever component of behavioral factors which are uncorrelated with SES, but are correlated with the distribution of students between classes. I.e., there will need to be a systematic reason why rich white kids who are more poorly behaved than the average rich white kid go to classroom 101 but not classroom 201. (If they are uncorrelated with the distribution between classes, they will merely contribute to sampling error. See my response to jbooth illustrating how the law of large numbers makes errors like this a small problem. Feel free to plug in your own numbers.)
    - v21 16y ago
      The only component of student behavior the predictor will not account for is whatever component of behavioral factors which are uncorrelated with SES, but are correlated with the distribution of students between classes. I.e., there will need to be a systematic reason why rich white kids who are more poorly behaved than the average rich white kid go to classroom 101 but not classroom 201. The root cause is that they share a classroom with other students who are misbehaving. To oversimplify wildly, take 4 demographically matched classes: 1) A class with some disruptive students , with a bad teacher : outcome - bad 2) A class with entirely generally well-behaved students, with a bad teacher : outcome - not so good 3) A class with some disruptive students, with a good teacher: outcome - not so good 4) A class with entirely generally well-behaved students, with a good teacher : outcome - good How do you distinguish between classes 2 and 3?
      - yummyfajitas 16y ago
        Here are two more or less obvious ways. 1) Average over the set of classes taught by any given teacher, maybe drop best/worst (you know, stats 101 stuff). Simple assumptions: do teacher evaluations once per year, 4 classes per semester (so data from 8 classes to evaluate teacher). Assume average scores of 80, and disruptive students reduce scores by 30, bad teachers reduce scores only by 10 pts. Assume disruptive students uncommon, unlucky teacher gets 1 class with disruptive students. Case 2) scores [80,80,80,50,80,80,80,80], mean=76. Mean drop top/bottom = 80. Case 3) scores [70, 70, 70, 70, 70, 70, 70, 70], mean=70. Mean drop top/bottom = 70. Assume disruptive students are common, unlucky good teacher gets 4 classes with disruptive students, lucky bad teacher gets 3. Case 2) Scores [80, 80, 80, 80, 50, 50, 50, 50], mean 65, MDTB=65. Case 3) Scores [ 70, 70, 70, 70, 70, 40, 40, 40], mean 58.76, MDTB=60. Assuming gaps of 5-10pts are statistically insignificant for one year, you occasionally are unable to distinguish between good and bad teachers in a single year. So every 2-3 years, you to a multi-year review. Now your sample size is up to 24 (from 8), and most likely both the good teacher and bad teacher have both had a lucky and unlucky class or two. 2) Include disciplinary problems in the predictor. Thus, the error is reduced from a class with some disruptive students (happens occasionally) to a class with some students who became disruptive for the first time ever (happens 1/12 as often).

by sethg 16y ago

yummyfajitas 16y ago

P < 0.05? That's terrible! How will the education system survive? On the other hand, lets compare it to our current system of ignoring teacher quality. Assuming 25% of teachers are significantly above average and 25% are below (anyone have data on this?), that's a 50% error rate. Let me just address your main statistical misconception, however: You can’t assume that after all the measurable factors in your statistical model are taken into account, the only remaining input to performance is the skill of the teacher. This is not an assumption. The assumption is that after all the measurable factors are taken into account, the remainder are unbiased (i.e., have mean 0) or at least have bias smaller than ignoring all data. Can you hypothesize an external factor which would, in a single measurement period (either a semester or year), significantly reduce the scores of 20-30 students of a single teacher while not affecting the scores of other demographically similar students? And do you really believe this occurs so often that it would make measurement worse than ignoring all data?

sethg 16y ago
“Either use Value Added Modeling or ignore all data” is a false dichotomy.
v21 16y ago
> Can you hypothesize an external factor which would, in a single measurement period (either a semester or year), significantly reduce the scores of 20-30 students of a single teacher while not affecting the scores of other demographically similar students? Yeah. Classes, like any small social group of people, have their own dynamics. While the teacher should be able to exercise some control and restraint, they'll obviously be less successful as teachers if that's where their energy is going. Whereas a well-behaved class, they can focus all their energy on teaching. If you adjust for the socioeconomic factors of each student individually, you can separate them out, but you can't separate out the overall dynamics of the group from the performance of the teacher.
- yummyfajitas 16y ago
  If you adjust for the socioeconomic factors of each student individually, you can separate them out... This is precisely what VAM does. Each student gets a predicted grade and they are combined to get an overall class grade. If things like well-behavedness are correlated with socioeconomic status, they will be incorporated into the predictor. Thus, a low SES class will have a lower predicted grade, and their poor behavior will be part of the reason why. The only component of student behavior the predictor will not account for is whatever component of behavioral factors which are uncorrelated with SES, but are correlated with the distribution of students between classes. I.e., there will need to be a systematic reason why rich white kids who are more poorly behaved than the average rich white kid go to classroom 101 but not classroom 201. (If they are uncorrelated with the distribution between classes, they will merely contribute to sampling error. See my response to jbooth illustrating how the law of large numbers makes errors like this a small problem. Feel free to plug in your own numbers.)
  - v21 16y ago
    The only component of student behavior the predictor will not account for is whatever component of behavioral factors which are uncorrelated with SES, but are correlated with the distribution of students between classes. I.e., there will need to be a systematic reason why rich white kids who are more poorly behaved than the average rich white kid go to classroom 101 but not classroom 201. The root cause is that they share a classroom with other students who are misbehaving. To oversimplify wildly, take 4 demographically matched classes: 1) A class with some disruptive students , with a bad teacher : outcome - bad 2) A class with entirely generally well-behaved students, with a bad teacher : outcome - not so good 3) A class with some disruptive students, with a good teacher: outcome - not so good 4) A class with entirely generally well-behaved students, with a good teacher : outcome - good How do you distinguish between classes 2 and 3?
    - yummyfajitas 16y ago
      Here are two more or less obvious ways. 1) Average over the set of classes taught by any given teacher, maybe drop best/worst (you know, stats 101 stuff). Simple assumptions: do teacher evaluations once per year, 4 classes per semester (so data from 8 classes to evaluate teacher). Assume average scores of 80, and disruptive students reduce scores by 30, bad teachers reduce scores only by 10 pts. Assume disruptive students uncommon, unlucky teacher gets 1 class with disruptive students. Case 2) scores [80,80,80,50,80,80,80,80], mean=76. Mean drop top/bottom = 80. Case 3) scores [70, 70, 70, 70, 70, 70, 70, 70], mean=70. Mean drop top/bottom = 70. Assume disruptive students are common, unlucky good teacher gets 4 classes with disruptive students, lucky bad teacher gets 3. Case 2) Scores [80, 80, 80, 80, 50, 50, 50, 50], mean 65, MDTB=65. Case 3) Scores [ 70, 70, 70, 70, 70, 40, 40, 40], mean 58.76, MDTB=60. Assuming gaps of 5-10pts are statistically insignificant for one year, you occasionally are unable to distinguish between good and bad teachers in a single year. So every 2-3 years, you to a multi-year review. Now your sample size is up to 24 (from 8), and most likely both the good teacher and bad teacher have both had a lucky and unlucky class or two. 2) Include disciplinary problems in the predictor. Thus, the error is reduced from a class with some disruptive students (happens occasionally) to a class with some students who became disruptive for the first time ever (happens 1/12 as often).