4 ms·

High variance absolutely implies that VAM doesn't work. If you are firing good teachers at random then only morons would go into teaching. So only a moron woul

High variance absolutely implies that VAM doesn't work. If you are firing good teachers at random then only morons would go into teaching. So only a moron would trade stocks, speculate on real estate, or take a job as a salesman? Of course, we just agreed that even if this is true, it doesn't matter. Morons are nearly as good at teaching as anyone else, and the results of good teaching fades over time anyway. So what would be the harm? It's a simple fact of statistics - the bigger the effect size, the easier it is to measure. It's simply innumeracy to claim large effects sizes and also impossibility to measure them. Why all the left wing mathematics denialism? (Another example of left wing mathematics denialism applies to the pigeon hole principle. If you have N houses and K > N people, K - N people won't have a house.)

Alex3917 10y ago
> So only a moron would trade stocks, speculate on real estate, or take a job as a salesman? All of those are examples of jobs with fairly to extremely reliable performance measurements... E.g. if a salesperson sells makes 20 sales in a quarter, then there is zero chance of them getting fired because their managers think they actually made -7 sales. > Morons are nearly as good at teaching as anyone else, and the results of good teaching fades over time anyway. Again, not accurate. The evidence that good teachers have some magical impact on student performance as measured by standardized test scores is flawed, but that doesn't have any relevance to the question of whether or not good/bad teachers exist in reality. And in fact, if you buy into the idea that "education isn't the filling of a vessel, but the lighting of a fire," then you'd expect the impact of good teachers to become larger over time. (But again, not as measured by VAM.)
- yummyfajitas 10y ago
  All of those are examples of jobs with fairly to extremely reliable performance measurements... E.g. if a salesperson sells makes 20 sales in a quarter, then there is zero chance of them getting fired because their managers think they actually made -7 sales. Whoah, suddenly you are applying a much lower level of skepticism to performance measurements now that we aren't talking about education. As you know, the mean standardized test score of students in a class (with no attempt to adjust for student quality) is just as objectively known as the number of sales. So I guess if we just did that, we'd be fine? Of course, the relevant question is how does that compare to a baseline # of sales that this salesperson could be expected to get? I knew a salesgirl doing enterprise software. In a given month she might make 0 sales, or 1, in a very good month 2. Is she a bad salesgirl because she had a month with 0 sales? That's variance, which according to you 100% invalidates a performance metric. You are making isolated demands for rigour in education that you don't seem to apply elsewhere. See this article for a description of this fallacy: http://slatestarcodex.com/2014/08/14/beware-isolated-demands-for-rigor/ http://slatestarcodex.com/2014/08/14/beware-isolated-demands... Now ask yourself, why are you doing this? Since you seem to believe good teachers exist, what evidence do you have of this? The lighting of fires in the belly of invisible dragons? http://www.godlessgeeks.com/LINKS/Dragon.htm http://www.godlessgeeks.com/LINKS/Dragon.htm
  - Alex3917 10y ago
    Teachers aren't ranked based on mean test scores, they're ranked based on their estimated contribution to student test scores. There are two issues here: 1) How accurate are the metrics 2) how stable are the metrics over time. Here are the differences between sales and teaching: - With VAM, not only do you have stability to worry about, but you also have the question of whether or not VAM is an accurate measure of teacher contribution to student performance in the first place. With sales you might argue that the number of sales or total contribution metrics don't accurately predict CLV or something, but the ambiguity there is trivial compared with VAM. - With respect to stability, with teaching you only get test scores once a year, which means that it takes about five years until the VAM scores of teachers become relatively stable. Whereas with sales people, you get new metrics every quarter. - Since what matters the most in business is having enough cashflow to stay in business, the short term view matters more than the longterm view. Whereas with students, what they achieve five and ten years out is much more important than how they perform at the end of each year. - With sales, there is zero barrier to entry, and if you get fired then you can just get another job two weeks later. With teaching you need to spend 2+ years and tens of thousands of dollars to get into the profession, and if you get fired then the best case scenario is that it takes an entire year to get another job. But often you're basically just banned from the profession. - With sales you're also making several times more money than you would as a teacher.
    - yummyfajitas 10y ago
      If teachers were estimated on mean test scores, we could rank them the same way we often rank salespeople. Would you be ok with this? If not, why not? And more importantly, why don't your newfound rationalizations apply equally well to salespeople or traders? (Incidentally, traders also are measured against a statistical model - risk adjustments + benchmark rate.) VAM is an accurate measure of teacher contribution to student performance in the first place. And with sales, you need to determine whether your attribution/commission model is an accurate measure of salesperson compensation to company profits. It's the exact same problem, it uses similar statistical methods, and it has variance. * Whereas with sales people, you get new metrics every quarter.* So test every quarter and evaluate on that basis. Done. Further, many salespeople have less information per month than teachers - for example, the enterprise salesgirl I mentioned earlier who has 0, 1 or 2 sales/month. You seem to be desperately reaching for rationalizations that explain why teaching is fundamentally different than every other profession. Why is that?
wfo 10y ago
The risk vs. reward profile for these jobs is very different. A stock trader can be successful one year and make up for losses the next, so long as he is successful more often than not. If an excellent teacher has a 25% chance of being fired every evaluation session, he has nothing to compensate for this risk -- just low pay, unconscionable hours, and a very difficult, thankless job. Most teachers are teachers because of belief in helping children and personal dedication. Flipping a coin and saying "you're fired if I get two heads in a row" every year means that is gone. It is not innumeracy to suggest there are so many confounding variables that are nearly impossible to separate from the treatment that it isn't a realistic or effective method for making real decisions about performance that negatively impact the careers and lives of dedicated public servants. It is innumeracy to suggest that "the bigger the effect size, the easier it is to measure" -- it is not incorrect technically, but it is very misleading. No matter how large the effect size, it can be very difficult to separate the effect of different variables with limited data (which is always the case -- we are testing one teacher against another) Granted, I haven't read those particular sources and I'm not sure if this is the approach they take. Of course phrases like "left wing mathematics denialism" are purposefully incendiary, contentless, and laughably absurd -- they belong somewhere like Breitbart, not here. Most Mathematicians are left-wing, for the record. I've never heard of anyone denying the pigeon hole principle in that context (or any context), but straw men are a very effective rhetorical device.
- yummyfajitas 10y ago
  A stock trader can be successful one year and make up for losses the next, so long as he is successful more often than not If you don't get good returns your first couple of years, you'll likely be out permanently. A couple of bad years on a 20 year trading record, you might be fine. Then again, Alex's sources all note that VAM stabilizes after a number of years (between 3 and 5). So long term teachers should be fine too. Similarly for salespeople. Like it or not, getting a professional evaluation based on a noisy objective measurement is nothing special. It happens in many professions - why should teachers be protected? Also, I didn't realize 38.5 hours/week, 9 months/year was "unconscionable". http://www.bls.gov/opub/mlr/2008/03/art4full.pdf http://www.bls.gov/opub/mlr/2008/03/art4full.pdf Also, teaching is not "thankless". Teachers are given a high degree of respect - #2 below nurses, in terms of ethics/honesty. http://www.gallup.com/file/poll/166364/Honesty_and_Ethics_of_Professions_131216.pdf http://www.gallup.com/file/poll/166364/Honesty_and_Ethics_of... In terms of competence and warmth teachers also rank near the top on both axes. http://www.pnas.org/content/111/Supplement_4/13593.abstract?sid=70a3fb78-4496-48b1-af32-fe27b57550fd http://www.pnas.org/content/111/Supplement_4/13593.abstract?... Or just note the typical language used to describe them, e.g. "dedicated public servants". What makes you think teaching is "thankless"? I've never heard of anyone denying the pigeon hole principle in that context (or any context), but straw men are a very effective rhetorical device. Go listen to discussions of housing in SF.
justratsinacoat 10y ago
>(Another example of left wing mathematics denialism applies to the pigeon hole principle. If you have N houses and K > N people, K - N people won't have a house.) Can't win an argument because the facts aren't on your side? Bring up a red herring for a random insult! Why all the talk of left wing mathematics denialism? This is a perfect example of $GROUP_I_DISAGREE_WITH's typical meaningless sophistry!!

Alex3917 10y ago

> So only a moron would trade stocks, speculate on real estate, or take a job as a salesman? All of those are examples of jobs with fairly to extremely reliable performance measurements... E.g. if a salesperson sells makes 20 sales in a quarter, then there is zero chance of them getting fired because their managers think they actually made -7 sales. > Morons are nearly as good at teaching as anyone else, and the results of good teaching fades over time anyway. Again, not accurate. The evidence that good teachers have some magical impact on student performance as measured by standardized test scores is flawed, but that doesn't have any relevance to the question of whether or not good/bad teachers exist in reality. And in fact, if you buy into the idea that "education isn't the filling of a vessel, but the lighting of a fire," then you'd expect the impact of good teachers to become larger over time. (But again, not as measured by VAM.)

yummyfajitas 10y ago
All of those are examples of jobs with fairly to extremely reliable performance measurements... E.g. if a salesperson sells makes 20 sales in a quarter, then there is zero chance of them getting fired because their managers think they actually made -7 sales. Whoah, suddenly you are applying a much lower level of skepticism to performance measurements now that we aren't talking about education. As you know, the mean standardized test score of students in a class (with no attempt to adjust for student quality) is just as objectively known as the number of sales. So I guess if we just did that, we'd be fine? Of course, the relevant question is how does that compare to a baseline # of sales that this salesperson could be expected to get? I knew a salesgirl doing enterprise software. In a given month she might make 0 sales, or 1, in a very good month 2. Is she a bad salesgirl because she had a month with 0 sales? That's variance, which according to you 100% invalidates a performance metric. You are making isolated demands for rigour in education that you don't seem to apply elsewhere. See this article for a description of this fallacy: http://slatestarcodex.com/2014/08/14/beware-isolated-demands-for-rigor/ http://slatestarcodex.com/2014/08/14/beware-isolated-demands... Now ask yourself, why are you doing this? Since you seem to believe good teachers exist, what evidence do you have of this? The lighting of fires in the belly of invisible dragons? http://www.godlessgeeks.com/LINKS/Dragon.htm http://www.godlessgeeks.com/LINKS/Dragon.htm
- Alex3917 10y ago
  Teachers aren't ranked based on mean test scores, they're ranked based on their estimated contribution to student test scores. There are two issues here: 1) How accurate are the metrics 2) how stable are the metrics over time. Here are the differences between sales and teaching: - With VAM, not only do you have stability to worry about, but you also have the question of whether or not VAM is an accurate measure of teacher contribution to student performance in the first place. With sales you might argue that the number of sales or total contribution metrics don't accurately predict CLV or something, but the ambiguity there is trivial compared with VAM. - With respect to stability, with teaching you only get test scores once a year, which means that it takes about five years until the VAM scores of teachers become relatively stable. Whereas with sales people, you get new metrics every quarter. - Since what matters the most in business is having enough cashflow to stay in business, the short term view matters more than the longterm view. Whereas with students, what they achieve five and ten years out is much more important than how they perform at the end of each year. - With sales, there is zero barrier to entry, and if you get fired then you can just get another job two weeks later. With teaching you need to spend 2+ years and tens of thousands of dollars to get into the profession, and if you get fired then the best case scenario is that it takes an entire year to get another job. But often you're basically just banned from the profession. - With sales you're also making several times more money than you would as a teacher.
  - yummyfajitas 10y ago
    If teachers were estimated on mean test scores, we could rank them the same way we often rank salespeople. Would you be ok with this? If not, why not? And more importantly, why don't your newfound rationalizations apply equally well to salespeople or traders? (Incidentally, traders also are measured against a statistical model - risk adjustments + benchmark rate.) VAM is an accurate measure of teacher contribution to student performance in the first place. And with sales, you need to determine whether your attribution/commission model is an accurate measure of salesperson compensation to company profits. It's the exact same problem, it uses similar statistical methods, and it has variance. * Whereas with sales people, you get new metrics every quarter.* So test every quarter and evaluate on that basis. Done. Further, many salespeople have less information per month than teachers - for example, the enterprise salesgirl I mentioned earlier who has 0, 1 or 2 sales/month. You seem to be desperately reaching for rationalizations that explain why teaching is fundamentally different than every other profession. Why is that?

wfo 10y ago

The risk vs. reward profile for these jobs is very different. A stock trader can be successful one year and make up for losses the next, so long as he is successful more often than not. If an excellent teacher has a 25% chance of being fired every evaluation session, he has nothing to compensate for this risk -- just low pay, unconscionable hours, and a very difficult, thankless job. Most teachers are teachers because of belief in helping children and personal dedication. Flipping a coin and saying "you're fired if I get two heads in a row" every year means that is gone. It is not innumeracy to suggest there are so many confounding variables that are nearly impossible to separate from the treatment that it isn't a realistic or effective method for making real decisions about performance that negatively impact the careers and lives of dedicated public servants. It is innumeracy to suggest that "the bigger the effect size, the easier it is to measure" -- it is not incorrect technically, but it is very misleading. No matter how large the effect size, it can be very difficult to separate the effect of different variables with limited data (which is always the case -- we are testing one teacher against another) Granted, I haven't read those particular sources and I'm not sure if this is the approach they take. Of course phrases like "left wing mathematics denialism" are purposefully incendiary, contentless, and laughably absurd -- they belong somewhere like Breitbart, not here. Most Mathematicians are left-wing, for the record. I've never heard of anyone denying the pigeon hole principle in that context (or any context), but straw men are a very effective rhetorical device.

yummyfajitas 10y ago
A stock trader can be successful one year and make up for losses the next, so long as he is successful more often than not If you don't get good returns your first couple of years, you'll likely be out permanently. A couple of bad years on a 20 year trading record, you might be fine. Then again, Alex's sources all note that VAM stabilizes after a number of years (between 3 and 5). So long term teachers should be fine too. Similarly for salespeople. Like it or not, getting a professional evaluation based on a noisy objective measurement is nothing special. It happens in many professions - why should teachers be protected? Also, I didn't realize 38.5 hours/week, 9 months/year was "unconscionable". http://www.bls.gov/opub/mlr/2008/03/art4full.pdf http://www.bls.gov/opub/mlr/2008/03/art4full.pdf Also, teaching is not "thankless". Teachers are given a high degree of respect - #2 below nurses, in terms of ethics/honesty. http://www.gallup.com/file/poll/166364/Honesty_and_Ethics_of_Professions_131216.pdf http://www.gallup.com/file/poll/166364/Honesty_and_Ethics_of... In terms of competence and warmth teachers also rank near the top on both axes. http://www.pnas.org/content/111/Supplement_4/13593.abstract?sid=70a3fb78-4496-48b1-af32-fe27b57550fd http://www.pnas.org/content/111/Supplement_4/13593.abstract?... Or just note the typical language used to describe them, e.g. "dedicated public servants". What makes you think teaching is "thankless"? I've never heard of anyone denying the pigeon hole principle in that context (or any context), but straw men are a very effective rhetorical device. Go listen to discussions of housing in SF.

justratsinacoat 10y ago

>(Another example of left wing mathematics denialism applies to the pigeon hole principle. If you have N houses and K > N people, K - N people won't have a house.) Can't win an argument because the facts aren't on your side? Bring up a red herring for a random insult! Why all the talk of left wing mathematics denialism? This is a perfect example of $GROUP_I_DISAGREE_WITH's typical meaningless sophistry!!