5 ms·

Teacher effectiveness ratings, programmers, and Khan Academy's data

55 points by kamens 15y ago

DarkShikari 15y ago
Teaching is a lot like programming in this respect -- if you try to enforce metrics of "performance", you usually end up with optimization for those metrics and dumber, less-educated students. Similarly, students tend to optimize too, just from a different standpoint. Examples of this: If you rate teachers based on the grades their students get, they will usually award their students higher grades for the same (or worse) performance. This isn't just a joke; many, many professors have reported their deans bearing down on them for awarding too many bad grades, often to students who are so incompetent that they cannot write basic English sentences or perform long division. The solution -- not remedial courses, or letting them drop out (that would hurt financial aid income) -- but letting them pass even though they lack the basic skills to learn in that class, let alone pass. This also leads to the student mentality that they are being "given" grades (and thus can beg for higher grades) as opposed to earning them through their own effort in the class. If you rate teachers based on the number of their students pass a standardized test, the teachers will focus solely on the contents of that standardized test and put a huge amount of effort into educating the few students who Don't Get It, at the cost of all the students that do. I had this experience in primary and secondary school, where teachers lamented that they had to go over material for the Standards of Learning tests constantly and repeatedly, dramatically lowering the amount they actually had time to teach. The end result is a quick dropping of everyone to the lowest common denominator -- which often just ends up shifting the entire bell curve to a lower point. If you rate teachers based on student evaluations, your results are nearly useless. Most students don't bother to write in much detail, since they're just glad to be done with the final exam (I did this too!)-- and often they use it for "revenge" on professors they thought were too hard. The teachers who get the best evaluations are often (though not always) the teachers who gave the easiest work and grades, and whose students turn out to be the most woefully unprepared for what comes next. Just like customers of your software are not always good judges of what they really want, students are often terrible judges of what good education actually is. As far as I can see, nobody has come up with a magical metric for rating teachers. If anything, most of the metrics out there are outright counterproductive -- and you can see it in schools across America, where teachers are being constantly urged to inflate grades for their 'customers', overlook plagiarism, ignore the most talented students in favor of the least talented ones, and help drive down the quality of education for the next generation.
- moonchrome 15y ago
  >As far as I can see, nobody has come up with a magical metric for rating teachers. If anything, most of the metrics out there are outright counterproductive -- and you can see it in schools across America, where teachers are being constantly urged to inflate grades for their 'customers', overlook plagiarism, ignore the most talented students in favor of the least talented ones, and help drive down the quality of education for the next generation. You can do things to improve the quality of data - use multiple metrics, don't announce them up front and certainly don't standardize - so that optimizing for a specific test doesn't justify the cost, also change testing as needed/with dynamic feedback and use the results to get insight in to the problem. Computers can only help with this sort of thing. Ultimately it's about defining the objective of education, what environment students are supposed to be prepared by it and exposing them to that environment, just like you want to test the system in production as soon as you can, benchmarks are always synthetic and can't replace "real world" testing. Data won't capture everything and could in fact be 100% misleading - that's the nature of imperfect knowledge - but if used correctly it can help provide insight in to situation. Just don't conflate benchmarks and reality, use it as a tool for improved feedback/analysis. Of course this approach is absolutely incompatible with bureaucratic systems. Heck the entire education system is based around optimizing for arbitrary/artificial evaluation metrics (all that crap about higher education and signaling, that starts with primary education). As for favoring least talented over most talented, that's just different objectives, I don't think it changes with more quantifiable information about students performance, do you want to raise the bottom at the expense of the top.
- yummyfajitas 15y ago
  If you rate teachers based on the number of their students pass a standardized test, the teachers will focus solely on the contents of that standardized test and put a huge amount of effort into educating the few students who Don't Get It, at the cost of all the students that do. Your issue here is not with standardized tests. Your issue is with the goals of the school system. I.e., you want schools to maximize mean(performance_data), but they are actually trying to maximize len([p for p in performance_data if p >= pmin]). Gathering good data is not the problem, it just reveals the problem.
crazygringo 15y ago
Of course, the real problem is crappy metrics. And this is why "teach to the test" is given such a bad rap. If the tests were actually well-written, testing all sorts of critical reasoning and problem-solving abilities, then teaching-to-the-test would be the correct strategy, and metrics would be tremendously useful. It makes me think somewhat of exams used for foreign students of English. The American TOEFL exam is extremely limited, and test prep can be extremely effective, without actually improving your day-to-day English. The British IELTS exam is extremely wide-ranging, and test prep is basically worthless, because it really does a good job of testing your English. To improve your test scores, you truly do have to improve your real-world English.
tristan_juricek 15y ago
I don't see a problem with effective metrics; they might help identify where teachers might need help. The problem, is that usually nobody really does proper statistical analysis. The metrics I've seen, are usually like "mean student grade performance". In which case, what you'd want to do is have a really small class of smart kids. (Ergo, kudos to the teacher who finds a way to dump the idiots.) I do think, however, you could do polling of teachers by asking the students, teachers, and other staff. In most cases you could easily have enough coworkers to figure out who's respected and who needs help.
deleted 15y ago
[deleted]
sbucher 15y ago
The author is making a straw man argument, though it does not seem to be intentional (edit: I am referring to Rubinstein's original analysis). The author points out that using data for a single class and single year is not a reliable indicator (second graph). That is uncontested and exactly why ratings are based on 3 years of data and multiple classes where possible. Even this cannot provide a single, accurate percentile value. That is why confidence intervals are used and are rather prominently displayed in all the graphs; e.g. <http://www.nytimes.com/schoolbook/school/656-ps-009-teunis-g-bergen/teachers&#62 http://www.nytimes.com/schoolbook/school/656-ps-009-teunis-g...; Given the example above, there is one teacher who has is 50th percentile for career math, but the confidence interval (CI) indicates that this may really be anywhere from well below average to well above average. In contrast there is another teacher with a value of 3 and the highest bounds of the CI still place them well below average. Conversely there is a teacher that is 90th pctl and even the lowest bounds of the the confidence interval still makes this a highly effective teacher. In sum, the data make evident that a few teachers are fairly unambiguously ineffective (at least with regards to test results), a few are unambiguously effective, but for the majority the most that can be said is that they are neither the best nor the worst, but somewhere in the middle. So if you reserve yourself to making conclusions merited by the data, then there is some value to this (i.e. identifying the extremely ineffective and effective). As it stands now, the teacher with a 3 may very well have tenure and be paid considerably more that the teacher with a 90 (given that pay is a function largely of seniority and education, and nearly all teachers have tenure). That is what this exercise is meant to address. So you definitely cannot assign a fine-grained single value to every teacher (and the authors analyses speak to that issue), but there does not mean you cannot make any conclusions at all. If folks make conclusions that exceed what is supported by the data that is a fault of the analyst or perhaps a function of poor communication or visualization, not evidence that the data is junk. That said, the test could be improved (and supposedly are being improved) and the very least the measurements place more focus on the subjects being tested. And the teacher effectiveness data is only 40% of the new approach to evaluating teachers in NY; the other 60% including peer evaluation. It may be useful if that became part of the public data so a more balanced picture is available; but it is unclear if that data would be public.
- kamens 15y ago
yummyfajitas 15y ago
I'm looking at the original posts that inspired this: http://garyrubinstein.teachforus.org/2012/02/26/analyzing-released-nyc-value-added-data-part-1/ http://garyrubinstein.teachforus.org/2012/02/26/analyzing-re... http://garyrubinstein.teachforus.org/2012/02/28/analyzing-released-nyc-value-added-data-part-2/ http://garyrubinstein.teachforus.org/2012/02/28/analyzing-re... In particular, the "same grade, different subjects" plot looks like a good correlation. Some noise, but otherwise exactly what you'd expect. Anyone know where to get the raw data so I can do more careful plots (e.g., a density plot instead of a scatter plot [1])? Note that noise simply means you need more than a single year's data to make a decision. [1] In a scatterplot, if two points overlap (which they likely do in this data), the apparent density is understated. [edit: found the data here: http://www.ny1.com/content/top_stories/156599/now-available--2007-2010-nyc-teacher-performance-data http://www.ny1.com/content/top_stories/156599/now-available-... ]
mathattack 15y ago
Any field that produces only one metric will find people optimizing on it. Eventually the metric loses it's value. This is true in teaching, programming, and most everything. Life's not so one dimensional.

55 points by kamens 15y ago

Teaching is a lot like programming in this respect -- if you try to enforce metrics of "performance", you usually end up with optimization for those metrics and dumber, less-educated students. Similarly, students tend to optimize too, just from a different standpoint. Examples of this: If you rate teachers based on the grades their students get, they will usually award their students higher grades for the same (or worse) performance. This isn't just a joke; many, many professors have reported their deans bearing down on them for awarding too many bad grades, often to students who are so incompetent that they cannot write basic English sentences or perform long division. The solution -- not remedial courses, or letting them drop out (that would hurt financial aid income) -- but letting them pass even though they lack the basic skills to learn in that class, let alone pass. This also leads to the student mentality that they are being "given" grades (and thus can beg for higher grades) as opposed to earning them through their own effort in the class. If you rate teachers based on the number of their students pass a standardized test, the teachers will focus solely on the contents of that standardized test and put a huge amount of effort into educating the few students who Don't Get It, at the cost of all the students that do. I had this experience in primary and secondary school, where teachers lamented that they had to go over material for the Standards of Learning tests constantly and repeatedly, dramatically lowering the amount they actually had time to teach. The end result is a quick dropping of everyone to the lowest common denominator -- which often just ends up shifting the entire bell curve to a lower point. If you rate teachers based on student evaluations, your results are nearly useless. Most students don't bother to write in much detail, since they're just glad to be done with the final exam (I did this too!)-- and often they use it for "revenge" on professors they thought were too hard. The teachers who get the best evaluations are often (though not always) the teachers who gave the easiest work and grades, and whose students turn out to be the most woefully unprepared for what comes next. Just like customers of your software are not always good judges of what they really want, students are often terrible judges of what good education actually is. As far as I can see, nobody has come up with a magical metric for rating teachers. If anything, most of the metrics out there are outright counterproductive -- and you can see it in schools across America, where teachers are being constantly urged to inflate grades for their 'customers', overlook plagiarism, ignore the most talented students in favor of the least talented ones, and help drive down the quality of education for the next generation.

moonchrome 15y ago
>As far as I can see, nobody has come up with a magical metric for rating teachers. If anything, most of the metrics out there are outright counterproductive -- and you can see it in schools across America, where teachers are being constantly urged to inflate grades for their 'customers', overlook plagiarism, ignore the most talented students in favor of the least talented ones, and help drive down the quality of education for the next generation. You can do things to improve the quality of data - use multiple metrics, don't announce them up front and certainly don't standardize - so that optimizing for a specific test doesn't justify the cost, also change testing as needed/with dynamic feedback and use the results to get insight in to the problem. Computers can only help with this sort of thing. Ultimately it's about defining the objective of education, what environment students are supposed to be prepared by it and exposing them to that environment, just like you want to test the system in production as soon as you can, benchmarks are always synthetic and can't replace "real world" testing. Data won't capture everything and could in fact be 100% misleading - that's the nature of imperfect knowledge - but if used correctly it can help provide insight in to situation. Just don't conflate benchmarks and reality, use it as a tool for improved feedback/analysis. Of course this approach is absolutely incompatible with bureaucratic systems. Heck the entire education system is based around optimizing for arbitrary/artificial evaluation metrics (all that crap about higher education and signaling, that starts with primary education). As for favoring least talented over most talented, that's just different objectives, I don't think it changes with more quantifiable information about students performance, do you want to raise the bottom at the expense of the top.
yummyfajitas 15y ago
If you rate teachers based on the number of their students pass a standardized test, the teachers will focus solely on the contents of that standardized test and put a huge amount of effort into educating the few students who Don't Get It, at the cost of all the students that do. Your issue here is not with standardized tests. Your issue is with the goals of the school system. I.e., you want schools to maximize mean(performance_data), but they are actually trying to maximize len([p for p in performance_data if p >= pmin]). Gathering good data is not the problem, it just reveals the problem.

crazygringo 15y ago

Of course, the real problem is crappy metrics. And this is why "teach to the test" is given such a bad rap. If the tests were actually well-written, testing all sorts of critical reasoning and problem-solving abilities, then teaching-to-the-test would be the correct strategy, and metrics would be tremendously useful. It makes me think somewhat of exams used for foreign students of English. The American TOEFL exam is extremely limited, and test prep can be extremely effective, without actually improving your day-to-day English. The British IELTS exam is extremely wide-ranging, and test prep is basically worthless, because it really does a good job of testing your English. To improve your test scores, you truly do have to improve your real-world English.

tristan_juricek 15y ago

I don't see a problem with effective metrics; they might help identify where teachers might need help. The problem, is that usually nobody really does proper statistical analysis. The metrics I've seen, are usually like "mean student grade performance". In which case, what you'd want to do is have a really small class of smart kids. (Ergo, kudos to the teacher who finds a way to dump the idiots.) I do think, however, you could do polling of teachers by asking the students, teachers, and other staff. In most cases you could easily have enough coworkers to figure out who's respected and who needs help.

deleted 15y ago

[deleted]

sbucher 15y ago

The author is making a straw man argument, though it does not seem to be intentional (edit: I am referring to Rubinstein's original analysis). The author points out that using data for a single class and single year is not a reliable indicator (second graph). That is uncontested and exactly why ratings are based on 3 years of data and multiple classes where possible. Even this cannot provide a single, accurate percentile value. That is why confidence intervals are used and are rather prominently displayed in all the graphs; e.g. <http://www.nytimes.com/schoolbook/school/656-ps-009-teunis-g-bergen/teachers&#62 http://www.nytimes.com/schoolbook/school/656-ps-009-teunis-g...; Given the example above, there is one teacher who has is 50th percentile for career math, but the confidence interval (CI) indicates that this may really be anywhere from well below average to well above average. In contrast there is another teacher with a value of 3 and the highest bounds of the CI still place them well below average. Conversely there is a teacher that is 90th pctl and even the lowest bounds of the the confidence interval still makes this a highly effective teacher. In sum, the data make evident that a few teachers are fairly unambiguously ineffective (at least with regards to test results), a few are unambiguously effective, but for the majority the most that can be said is that they are neither the best nor the worst, but somewhere in the middle. So if you reserve yourself to making conclusions merited by the data, then there is some value to this (i.e. identifying the extremely ineffective and effective). As it stands now, the teacher with a 3 may very well have tenure and be paid considerably more that the teacher with a 90 (given that pay is a function largely of seniority and education, and nearly all teachers have tenure). That is what this exercise is meant to address. So you definitely cannot assign a fine-grained single value to every teacher (and the authors analyses speak to that issue), but there does not mean you cannot make any conclusions at all. If folks make conclusions that exceed what is supported by the data that is a fault of the analyst or perhaps a function of poor communication or visualization, not evidence that the data is junk. That said, the test could be improved (and supposedly are being improved) and the very least the measurements place more focus on the subjects being tested. And the teacher effectiveness data is only 40% of the new approach to evaluating teachers in NY; the other 60% including peer evaluation. It may be useful if that became part of the public data so a more balanced picture is available; but it is unclear if that data would be public.

kamens 15y ago

yummyfajitas 15y ago

I'm looking at the original posts that inspired this: http://garyrubinstein.teachforus.org/2012/02/26/analyzing-released-nyc-value-added-data-part-1/ http://garyrubinstein.teachforus.org/2012/02/26/analyzing-re... http://garyrubinstein.teachforus.org/2012/02/28/analyzing-released-nyc-value-added-data-part-2/ http://garyrubinstein.teachforus.org/2012/02/28/analyzing-re... In particular, the "same grade, different subjects" plot looks like a good correlation. Some noise, but otherwise exactly what you'd expect. Anyone know where to get the raw data so I can do more careful plots (e.g., a density plot instead of a scatter plot [1])? Note that noise simply means you need more than a single year's data to make a decision. [1] In a scatterplot, if two points overlap (which they likely do in this data), the apparent density is understated. [edit: found the data here: http://www.ny1.com/content/top_stories/156599/now-available--2007-2010-nyc-teacher-performance-data http://www.ny1.com/content/top_stories/156599/now-available-... ]

mathattack 15y ago

Any field that produces only one metric will find people optimizing on it. Eventually the metric loses it's value. This is true in teaching, programming, and most everything. Life's not so one dimensional.