7 ms·

In deteriming whether course evaluations are corrlelated with teacher quality, they compared them to grades received in a follow on course, and found that they

In deteriming whether course evaluations are corrlelated with teacher quality, they compared them to grades received in a follow on course, and found that they had a negative correlation. So far so good. But if grades received in follow on courses are the gold standard for teacher effectiveness, then why not use that, instead of the other proxies the article goes on to recommend (peer evaluations primarily)? Certainly it won't be possible in every case as not every class has a follow on, but at the very least they should subject their new proposed methods to the same analysis.

Alex3917 12y ago
> if grades received in follow on courses are the gold standard for teacher effectiveness, then why not use that Because grades received in later courses isn't actually a meaningful metric. There was a theory that you could measure teacher quality this way using something called Value Added Metrics (VAM), but it's now been completely debunked. C.f. http://www.epi.org/publication/bp278/ http://www.epi.org/publication/bp278/ http://www.nber.org/papers/w14442 http://www.nber.org/papers/w14442 http://annenberginstitute.org/sites/default/files/product/211/files/valueAddedReport.pdf http://annenberginstitute.org/sites/default/files/product/21... http://ies.ed.gov/ncee/pubs/20104004/pdf/20104004.pdf http://ies.ed.gov/ncee/pubs/20104004/pdf/20104004.pdf http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1142237 http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1142237 http://educationnext.org/files/ednext20022_10.pdf http://educationnext.org/files/ednext20022_10.pdf
- yummyfajitas 12y ago
  Your links do not remotely come close to debunking the concept of VAM. Your first (accurately) says "measurements are noisy" and the inaccurately says "and other things matter for learning too". Unless I missed something, it doesn't provide a reason to expect bias in the other things. The second link says that some VAM models assume student assignment is random when it is not, and also that measured educational outputs don't matter much for outcomes anyway. The logical conclusion of this is simply that VAM needs modification, and maybe we should reduce education. Claiming this debunks the idea of VAM is like saying "hah, this paper proves OLS doesn't always work well, therefore regression is debunked". The third is really long and I couldn't figure out it's point. The fourth one says VAM is noisy. If you believe VAM is unable to measure the effects of teachers, are you willing to follow through with the logical conclusions? Namely, the effect teachers have on students is so small it can't be measured and we should therefore focus on other things (e.g. reducing cost) instead?
  - Alex3917 12y ago
    > Unless I missed something, it doesn't provide a reason to expect bias in the other things. "VAM’s instability can result from differences in the characteristics of students assigned to particular teachers in a particular year, from small samples of students (made even less representative in schools serving disadvantaged students by high rates of student mobility), from other influences on student learning both inside and outside school, and from tests that are poorly lined up with the curriculum teachers are expected to cover, or that do not measure the full range of achievement of students in the class." > If you believe VAM is unable to measure the effects of teachers, are you willing to follow through with the logical conclusions? Namely, the effect teachers have on students is so small it can't be measured ... We already knew that the contributions of teachers to student achievement was relatively small long before VAM was invented. The only people who seem to think that teachers have a very large impact are the pro-VAM folks (e.g. Bill Gates and Malcolm Gladwell), who all cite the same flawed study. That said, the reasons why VAM can't measure teacher performance isn't because the contributions of teachers are small per se, but rather because there are all sorts of problems with all of the proposed measurement systems. In order to create a VAM system that actually worked, assuming such a thing is possible, you'd have to make a number of changes to the way that the school system is designed... e.g. assigning students to teachers at random, designing standardized tests to measure teacher quality, implementing standardized tests in all subject areas, etc. But implementing all of those changes may well be worse for student learning than firing all the bad teachers or whatever. Not to mention that there is currently a major crisis in teacher retention, with half of all teachers either getting fired or quitting in their first five years. (After spending two years and god knows how much money getting their masters degrees to go into the profession.)
    - yummyfajitas 12y ago
      VAM’s instability Your quote concerns variance, not bias. (The main exception is the bit about "full range of achievement". It's quite true that teachers of upper class Asian students will likely have lower VAM scores than they deserve since the test caps out at 100%.) We already knew that the contributions of teachers to student achievement was relatively small long before VAM was invented. You dodged the question. Do you think policymakers and politicians should stop trying to improve teachers, and instead simply try to reduce costs? I.e., pay teachers less, increase their turnover (so that we don't have to pay expensive but ineffective senior ones), etc?
alecdbrooks 12y ago
It wasn't just the measure, it was that the experiment was randomized. Peer evaluations can be done for courses without follow ons and are more useful without randomization. I agree that he should be more critical of his own recommendations. It's not clear that faculty would be free of the "gender, and ethnicity, and attractiveness" biases studies suggest students have -- I was hoping he'd have a clever way to blind evaluations.

Alex3917 12y ago

> if grades received in follow on courses are the gold standard for teacher effectiveness, then why not use that Because grades received in later courses isn't actually a meaningful metric. There was a theory that you could measure teacher quality this way using something called Value Added Metrics (VAM), but it's now been completely debunked. C.f. http://www.epi.org/publication/bp278/ http://www.epi.org/publication/bp278/ http://www.nber.org/papers/w14442 http://www.nber.org/papers/w14442 http://annenberginstitute.org/sites/default/files/product/211/files/valueAddedReport.pdf http://annenberginstitute.org/sites/default/files/product/21... http://ies.ed.gov/ncee/pubs/20104004/pdf/20104004.pdf http://ies.ed.gov/ncee/pubs/20104004/pdf/20104004.pdf http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1142237 http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1142237 http://educationnext.org/files/ednext20022_10.pdf http://educationnext.org/files/ednext20022_10.pdf

yummyfajitas 12y ago
Your links do not remotely come close to debunking the concept of VAM. Your first (accurately) says "measurements are noisy" and the inaccurately says "and other things matter for learning too". Unless I missed something, it doesn't provide a reason to expect bias in the other things. The second link says that some VAM models assume student assignment is random when it is not, and also that measured educational outputs don't matter much for outcomes anyway. The logical conclusion of this is simply that VAM needs modification, and maybe we should reduce education. Claiming this debunks the idea of VAM is like saying "hah, this paper proves OLS doesn't always work well, therefore regression is debunked". The third is really long and I couldn't figure out it's point. The fourth one says VAM is noisy. If you believe VAM is unable to measure the effects of teachers, are you willing to follow through with the logical conclusions? Namely, the effect teachers have on students is so small it can't be measured and we should therefore focus on other things (e.g. reducing cost) instead?
- Alex3917 12y ago
  > Unless I missed something, it doesn't provide a reason to expect bias in the other things. "VAM’s instability can result from differences in the characteristics of students assigned to particular teachers in a particular year, from small samples of students (made even less representative in schools serving disadvantaged students by high rates of student mobility), from other influences on student learning both inside and outside school, and from tests that are poorly lined up with the curriculum teachers are expected to cover, or that do not measure the full range of achievement of students in the class." > If you believe VAM is unable to measure the effects of teachers, are you willing to follow through with the logical conclusions? Namely, the effect teachers have on students is so small it can't be measured ... We already knew that the contributions of teachers to student achievement was relatively small long before VAM was invented. The only people who seem to think that teachers have a very large impact are the pro-VAM folks (e.g. Bill Gates and Malcolm Gladwell), who all cite the same flawed study. That said, the reasons why VAM can't measure teacher performance isn't because the contributions of teachers are small per se, but rather because there are all sorts of problems with all of the proposed measurement systems. In order to create a VAM system that actually worked, assuming such a thing is possible, you'd have to make a number of changes to the way that the school system is designed... e.g. assigning students to teachers at random, designing standardized tests to measure teacher quality, implementing standardized tests in all subject areas, etc. But implementing all of those changes may well be worse for student learning than firing all the bad teachers or whatever. Not to mention that there is currently a major crisis in teacher retention, with half of all teachers either getting fired or quitting in their first five years. (After spending two years and god knows how much money getting their masters degrees to go into the profession.)
  - yummyfajitas 12y ago
    VAM’s instability Your quote concerns variance, not bias. (The main exception is the bit about "full range of achievement". It's quite true that teachers of upper class Asian students will likely have lower VAM scores than they deserve since the test caps out at 100%.) We already knew that the contributions of teachers to student achievement was relatively small long before VAM was invented. You dodged the question. Do you think policymakers and politicians should stop trying to improve teachers, and instead simply try to reduce costs? I.e., pay teachers less, increase their turnover (so that we don't have to pay expensive but ineffective senior ones), etc?

alecdbrooks 12y ago

It wasn't just the measure, it was that the experiment was randomized. Peer evaluations can be done for courses without follow ons and are more useful without randomization. I agree that he should be more critical of his own recommendations. It's not clear that faculty would be free of the "gender, and ethnicity, and attractiveness" biases studies suggest students have -- I was hoping he'd have a clever way to blind evaluations.