The author is making a straw man argument, though it does not seem to be intentional (edit: I am referring to Rubinstein's original analysis). The author points

The author is making a straw man argument, though it does not seem to be intentional (edit: I am referring to Rubinstein's original analysis). The author points out that using data for a single class and single year is not a reliable indicator (second graph). That is uncontested and exactly why ratings are based on 3 years of data and multiple classes where possible. Even this cannot provide a single, accurate percentile value. That is why confidence intervals are used and are rather prominently displayed in all the graphs; e.g. <http://www.nytimes.com/schoolbook/school/656-ps-009-teunis-g-bergen/teachers&#62 http://www.nytimes.com/schoolbook/school/656-ps-009-teunis-g...; Given the example above, there is one teacher who has is 50th percentile for career math, but the confidence interval (CI) indicates that this may really be anywhere from well below average to well above average. In contrast there is another teacher with a value of 3 and the highest bounds of the CI still place them well below average. Conversely there is a teacher that is 90th pctl and even the lowest bounds of the the confidence interval still makes this a highly effective teacher. In sum, the data make evident that a few teachers are fairly unambiguously ineffective (at least with regards to test results), a few are unambiguously effective, but for the majority the most that can be said is that they are neither the best nor the worst, but somewhere in the middle. So if you reserve yourself to making conclusions merited by the data, then there is some value to this (i.e. identifying the extremely ineffective and effective). As it stands now, the teacher with a 3 may very well have tenure and be paid considerably more that the teacher with a 90 (given that pay is a function largely of seniority and education, and nearly all teachers have tenure). That is what this exercise is meant to address. So you definitely cannot assign a fine-grained single value to every teacher (and the authors analyses speak to that issue), but there does not mean you cannot make any conclusions at all. If folks make conclusions that exceed what is supported by the data that is a fault of the analyst or perhaps a function of poor communication or visualization, not evidence that the data is junk. That said, the test could be improved (and supposedly are being improved) and the very least the measurements place more focus on the subjects being tested. And the teacher effectiveness data is only 40% of the new approach to evaluating teachers in NY; the other 60% including peer evaluation. It may be useful if that became part of the public data so a more balanced picture is available; but it is unclear if that data would be public.