9 ms·

Gates Foundation study: We’ve figured out what makes a good teacher

77 points by eggspurt 14y ago

edtechdev 14y ago
This work is valuable, but I personally would like to put more focus on improving teaching and learning - helping teachers do better, helping students do better, creating better learning environments and curricula. For example the budget of this study came out to about $15000 per teacher. What if instead we spent $5000 of that for professional development / training for the teacher, $5000 for a classroom set of chromebooks, and gave the teacher a $5000 raise. That alone would cause some significant improvements, I believe (not that that is ALL that is needed, and there are so many bigger issues that need to be tackled as someone else mentioned, such as poverty and inequity). But when we talk about "good" teaching and "bad" teaching, we are treating it like it is a simple craft that is mastered or not mastered. Teaching is like engineering, it involves design, and design is about making things better (see this prezi on learning design: http://prezi.com/b44jwdgvs8nl/olds-mooc-introduction/ http://prezi.com/b44jwdgvs8nl/olds-mooc-introduction/ )
- jonknee 14y ago
  Because this was a study, the money spent has little to do with the individual subjects. If pollster calls me for a a national opinion poll I would rather be paid 1/1000 of whatever the budget was than answer the questions, but obviously my participation is more helpful overall. The goal was to identify what actually works, which is something you need to know before improving the situation. Paying a bad teacher more will not make the education provided any better (despite whatever the teacher's union says).
dspeyer 14y ago
If I'm reading the linked page correctly, these metrics were verified by correlating them with individual student improvement on a variety of standardized tests and "more cognitively challenging assignments in math and english". This is better than pure circular logic, but until those metrics are validated, these are suspect. In particular, everything seems to test for shallow understanding.
murbard2 14y ago
If this becomes policy, it will most likely become another example of Goodhart's law. http://en.wikipedia.org/wiki/Goodharts_law http://en.wikipedia.org/wiki/Goodharts_law You can find a measure that correlates to performance, but once you use it to actually evaluate teacher, teacher will adapt and the measure will lost all relevance. Rely on test scores? Teach the test, not the material. Rely on student evaluations? Bribe them with easy material and good grades. Rely on principal evaluation? Get on its good side.
arocks 14y ago
Actually they have figured out how to evaluate teachers based on statistical models based on their repeatability of success. Finding a 'great' teacher is much more than finding the top rated teacher in my opinion. What qualities makes such a teacher good is a totally different matter. The latter was what I was hoping to find.
aaronharnly 14y ago
I, like many, hope the MET project yields fruit. However, see this post for a strong critique of one of the key claims of the MET study: http://garyrubinstein.teachforus.org/2013/01/09/the-50-million-dollar-lie/ http://garyrubinstein.teachforus.org/2013/01/09/the-50-milli... Namely, in the scatterplot correlating predicted effectiveness and actual effectiveness, they pooled teachers into 5% groups. This naturally reduces the variance, yielding a much straighter line of points than would an ordinary scatter plot. This is not, to my knowledge, an accepted or ordinary technique.
- crusso 14y ago
  Ugh, right from the get-go, Rubinstein throws away any attempt at statistical rigor by using an outlier example to try to prove something. The 5% groupings doesn't evaluate the metric. It just means that if they plan to use this actual study as a guideline, they should use the same technique in grouping teachers.
- MarkMc 14y ago
  I don't know enough about statistics to judge the strength of Mr Rubinstein's critique, but I find this a little odd: The axes on his graph is a 'raw score' between -1 and 1, but the axes on the MET study graph is 'standard deviations'. Is that significant?
nathan_long 14y ago
I can't say whether the findings are correct or not, but I like this: >> The teachers who seemed to be effective were, in fact, able to repeat those successes with different students in different years, the researchers found. Their students not only scored well on standardized exams but also were able to handle more complicated tests of their conceptual math knowledge and reading and writing abilities. To everyone who keeps saying "standardized tests are bad", please cut it out. A particular code benchmark may be useless, but benchmarking overall is good. Existing standardized tests may be bad, but in general, having some way to make objective measurements and adjust accordingly is the basis of all human improvement. And yes, we should reward good performance, if we can determine what good performance is. How else do you expect to get more of it? It is incredibly difficult to make incentives that actually do what you want rather than encourage gaming of the system, but that doesn't mean we shouldn't try. In short, let's stop debating whether we should measure and incentivize, and start debating how.
- jgj 14y ago
  I'm not sure that the argument is even that existing standardized tests are bad so much as that they are given too much priority. They should not be the only or primary means by which teachers, students, districts, etc are benchmarked (and in turn, rewarded or disciplined). "Teaching the test" is the problem, not necessarily the tests themselves.
- UnoriginalGuy 14y ago
  I grew up in the UK in a school where the emphasis of our entire education was "the test." Teachers entirely forgot how to teach a subject and our whole time was taken up with what essentially amounted to "test prep." I did pass their tests, but did I get a good education? Frankly, no, I don't think I did. I mean statistically I got a great education, but it seems we have forgotten why we even educate to begin with... Do we educate so people look smart on some random league table? Or do we educate so people are actually more well rounded, have a better grasp of the world around them, and do better as adults?
  - crusso 14y ago
axelfreeman 14y ago
"We’ve figured out what makes a good teacher" They've figured out that a "good" teacher can repeat the process of success (whatever this is), there is no information how he made it. The headline is slightly misleading.
- jonknee 14y ago
  The good news is they have video of good teachers (and bad) at their craft for years. It will take more analysis, but they have all the data at the ready.
UnoriginalGuy 14y ago
Using test scores in order to judge the quality of a teacher is like using brush strokes in order to measure the quality of a painter.
- crusso 14y ago
  So where part of the study showed that with randomized redistributions of students, certain teachers were able reproduce higher test scores -- your countering evidence shows otherwise. Great. Where's your evidence? My own evidence (experience going through school and generalization of how ability works in most professions) tells me that the good teachers in my life helped me to learn the information better and more deeply. My memories of those classrooms are of the subject matter and my desire to achieve in that teacher's environment. The crappy ones were noticeably so in that my memories of those classrooms was of misbehaving with my friends, having no desire to help the teacher teach me, and cramming to learn material for tests because I hadn't absorbed anything beforehand.
  - sethammons 14y ago
    I just realized that I keep replying to you, crusso (lol). Cheers. I'm more interested in the demographics of the student population than the 'random distribution (1)' of students in that population to different teachers. What I mean by this is that I would expect 'good' teachers to be able to demonstrate their abilities with with relatively homogeneous groupings of students where those students are from relatively decent neighborhoods. If, however, this study included teachers in inner city schools with huge variance in the 'quality' of student that a teacher would receive at often vastly different levels of understanding in the teacher's discipline area, then I think the results would be interesting insofar as to allow others to dig into what makes these teachers effective. 1: If we are looking at a teacher who teaches a course with a pre-requisite, the grouping of students becomes more similar (ie, a trigonometry teacher is likely to have students both motivated to do well and to have passed the requisite lower level math classes, and their success is less surprising to me).
    - crusso 14y ago
cbsmith 14y ago
I'll have to look at the prior reports, but this 3rd report is somewhat disappointing in terms of its statistical rigor. They do provide weightings based on optimal predictions for maximum accuracy with predictions for state tests scores, although those weights appear to assume a simple linear correlation. Even then, the data suggests that in class observations are almost a complete waste of time (single digit weightings), and that student surveys decline in value as the kids mature (no shocker: with younger kids, how much they like the teacher has a much more profound impact on their success at learning). In all cases the kids seem to be a more reliable indicator than the observations of professionals! This may just be an indication that assessments based on more frequent observations are more accurate, but the only way we can really get a sense of that is if they also had peer observations from team teaching environments... which are completely absent. That's not the worst of it though. The worst is the rest of the data. It is based on some fairly arbitrary presupposed models for combining these different assessments. No breakdown of the individual components of these assessments (other than "English" & "Math") and no attempt to discern a model from first principles. There are some other factors as well. Aside from some outliers, the accuracy of predictions seems kind of suspect. Most of the outcomes are within +/-0.05 standard deviations, and a majority of them are within what looks like +/-0.025 standard deviations. These are pretty minuscule variances (0.25 was supposed to represent a year of schooling, so 0.025 represents maybe 15 school days worth of progress at best), and it is not hard to see that without some of those outliers, the regression line doesn't look so pretty (particularly for English). I also didn't see any metrics on the variance of the data within a classroom, which I suspect makes any predictive value look limited at best. Ugh. I really hoped that Gates would make sure there was more statistical rigor with an analysis like this. :-( I hate to say it, but the conclusions of the study are not well supported by at least what is in the final research report.
- wheaties 14y ago
  Yes, this is also the same foundation that spent 1 billion to make schools smaller, because they didn't do simple statistical analysis to find that smaller schools showed much larger variability. Now they're coming back to say "oops, we were wrong." http://marginalrevolution.com/marginalrevolution/2010/09/the-small-schools-myth.html http://marginalrevolution.com/marginalrevolution/2010/09/the...
madaxe 14y ago
"Test scores" Brilliant. Well, yes, that is the best way to measure a teacher's efficacy, if your goal is to teach students to jump through hoops and pass standardised tests rather than to think. The best teacher I ever had was my grandmother - she taught me the most important lessons I'd ever learn, such as: "Observe the world around you." "Reach your own conclusions." "What was fact yesterday may be fiction tomorrow." Tests are a distraction from learning.
- coob 14y ago
  Did you read the article? It didn't use just standardised test scores.
  - madaxe 14y ago
    Of course I did, but standardised tests, particularly in the US, do little other than teach people how to game multiple choice, and pass tests. Life isn't about passing tests, it's about learning.
- crusso 14y ago
  I don't really understand the visceral reaction against testing in this HN thread. You're tested throughout your working life. Almost every job interview is a test. Giving a customer presentation is a test. Finding bugs in your software for a release date is a test. Memorizing that big Cheesecake Factory menu so you can wait tables is a test. Getting your General Contractor's license is a test. Getting certified as a Pilot requires passing a test. There should be more to education besides testing, but being tested is a part of life and it's easy to measure success - that combination means that it's a useful tool for education.
  - Mvandenbergh 14y ago
    Job interviews are a good example of a kind of testing that we know doesn't work all that well but we keep around for lack of a better alternative. The others are not good examples because they are useful in-and-of themselves. Giving good presentations and finding bugs are useful because they are useful, not because they predict something else. Pilot's licenses, like driver's licenses are based on tests where you more or less do the thing that you're being tested on. The test and the thing that it's supposed to measure are very similar and we can therefore use it to predict performance. In cases like that "teaching to the test" doesn't exist because practicing for the activity and for the test is the same thing. I'm not denying that educational testing is useful to quantify progress but such tests are often different enough from the underlying activity they're intended to measure that effort spent preparing for the test has little other purpose.
gnosis 14y ago
There are many important factors which teachers (no matter how good they are) can't control: 1. How wealthy, poor, educated, or uneducated a child's parents are. 2. How many resources the school and teacher are given. 3. Whether a child's parents are abusing them at home. 4. Whether a child is witnessing violence, hopelessness, despair in their neighborhood. 5. The attitude of a child's family and friends towards education. 6. Whether a child's parents are active in their child's education and in a PTA. Study after study has shown that these factors greatly influence a child's attention and attitude to school work, their attendance, and ultimately their performance at school. No child's performance on a standardized test, and no evaluation of a teacher's performance is going to pick up on the above factors, yet they can critically impact how well the child ultimately learns and what the child learns. Standardized testing of a child that is unfortunate enough to live in a situation where the above factors are against them will likely notice a poorly performing student, one who gradually does worse and worse on the tests until they finally flunk out. Then, instead of getting extra aid, the funding to the school that has a lot of such children will be cut (thanks to programs like No Child Left Behind), and their teachers penalized. These factors, which are strongly associated with poverty, are things a given teacher is in no position to influence, but which society as a whole could address. If it has the will.
- Mvandenbergh 14y ago
  That's why they do value-added testing rather than just evaluating teachers based on the score of their pupils. Obviously there are confounding factors, but if a teacher can make a statistically significant difference to the the test scores - as measured by before and after testing then that is valuable to know.
- Symmetry 14y ago
  So, if a teacher has as little effect on a child's performance as you seem to be implying should we then reduce teacher pay, accept the lower quality applicants we would get since it doesn't matter, and spend the extra money on anti-poverty measures?
- crusso 14y ago
  And at my job, there are many factors out of my control. I don't control the home life of my engineers. I don't control my customers' decision making processes. I don't control my competitors' actions in the market. But yet I'm judged on the small slice of that overall pie that I can control. What this study indicates is that there are teachers who can reproducibly create success levels given the control of the situation that they do have. How is it any different than the rest of the working world where you can control some things and you can't control others yet your job is to do the best you can with what you can control?
the_bear 14y ago
It's common for people who received good educations to say that test scores aren't an acceptable indication of learning. It's probably true that if you already have an acceptable level of education, test scores shouldn't be too important, but it seems like the people criticizing test scores haven't experienced what education is like for the children being "left behind." In middle school, the last three weeks of each semester were spent watching movies (the same two or three movies over and over). In high school, I was taught that the big bang theory was when the continents divided. I was taught that the most direct route between two points on the globe is through the north pole. My Spanish and woodworking teachers taught math because they knew that the math teachers weren't doing it. When you're talking about schools like the ones I went to (which weren't nearly as bad as many other schools in America) I think that test scores are a decent indicator of the education a student is receiving. You're measuring things like "can this kid read English" and "can they multiply two numbers together." Standardized tests are certainly capable of measuring things like that. Testing probably breaks down for measuring intelligence/education as the level of education improves, but I encourage everyone to consider just how dysfunctional many schools are before ignoring testing entirely.
- roguecoder 14y ago
  Except that even those things don't predict later success. Any kid with a family that can afford books will score higher on "can this kid read English". Any kid who's been drilled can learn to multiply two numbers together. Without cognitive skills behind them, though, those are not useful abilities. The best predictors of future success are self-control, emotional intelligence and a grasp of functional algebra (which I believe, but don't have evidence to support, is because of the importance of computational and systems-level thinking.) Only one of those, self-control, is even indirectly measured by current standardized tests.
  - ims 14y ago
    > Any kid with a family that can afford books will score higher on "can this kid read English". True, but there are also many families that can afford books but don't buy them, instead spending money on other things. Not having books in the house has been shown to correlate with decreased academic success.[1] I'm not saying this isn't a wealth issue, but that probably isn't the only dimension. [1] http://www.sciencedirect.com/science/article/pii/S0276562410000090 http://www.sciencedirect.com/science/article/pii/S0276562410...
brudgers 14y ago
The study identifies teachers who are effective against a particular metric. Most identified are likely members of the subset of good teachers. Yet, the three R's as measured by standardized test outcomes are not the only measure of effective education, just one that is cheap to measure. Dewey believed that the goal of education should be to produce good citizens. Systems with metrics based upon the model of student as consumer are more likely to produce outcomes biased toward producing consumers. The entire set of good teachers includes those who inspire a lifetime of learning. It includes those who teach kindness, sportsmanship, and good humor. It includes those who teach subjects for which there are few students and in which standardized tests make little sense such as art and shop. Maybe the study can point us in the right direction. But few districts will bring outside evaluators from Cambridge's famed institutions. Few districts will fund ETS analysis or purchase their tools. What we are likely to get are processes localized to the political reality on the ground.
robdoherty2 14y ago
I believe that studies like this are well-intentioned in that those conducting the research are truly trying to improve the state of education in the US. However, the result of their work I'm afraid is the 'quantification' of teaching as a profession.
jnazario 14y ago
kudos to the WaPo for giving a link to the study. too often, many publications discuss a study but never link to it, which then makes it harder to go and review it for yourself.
blueprint 14y ago
The title of this article, and perhaps the researchers' conclusions from their data, is wrong. They didn't find the criteria of a great teacher (that is, what makes a good teacher). They just found some correlations among the _qualities_ of good teachers. It's the total opposite meaning. They found results of good teachers, not causes.
loucal 14y ago
Is this the same gates foundation that I saw on 60 minutes pumping GMO seeds and fossil fuel fertilizer to third world countries to "help" them out of their food crisis? In fairness, I think I heard they realized something like "what works in america doesn't work everywhere" (bit of a cop out to save face) and moved toward more sustainable organic methods, but I'm not completely sure. All I know is, if its that hard for them to figure out the most practical way to grow a plant, we should probably take their advice on growing the minds of our children with a grain of salt.
- atomical 14y ago
  There's nothing wrong with using GMO seeds to increase yield.
  - loucal 14y ago
    It is opinion either way but in my opinion not only isn't it necessary, but it clearly threatens global biodiversity. Proper sustainable farming techniques produce an abundance and a variety of food, so the extra 1% increase on a huge monocropped field isn't even something that makes sense for third world communities. On the biodiversity thing, viewing your website leads me to believe you appreciate nature, so I'm just scratching my head that you haven't made the connection. Ohh well, downvote me all you want buddy. You are entitled to your opinion also.
    - atomical 14y ago
      It's much more than 1%. I haven't down voted you. Liking nature really has nothing to do with anything else.
MaysonL 14y ago
Evaluating teachers is all well and good, but the key to improving our educational system is not that. What is needed is improving teachers. See "Building a Better Teacher"[0] [0]http://www.nytimes.com/2010/03/07/magazine/07Teachers-t.html?ref=magazine&pagewanted=all&_r=0 http://www.nytimes.com/2010/03/07/magazine/07Teachers-t.html...
tokenadult 14y ago
The article kindly submitted here links to the project homepage http://www.metproject.org/ http://www.metproject.org/ at which the latest project report can be found. I've got some reading to do before I think about what else to add to the discussion already underway here. I get the impression that one observation that has motivated much of this research is how persons educated overseas have fared in United States society after immigrating here in early adulthood. My own experience living in another country (twice, in two different three-year stays) does much to prompt my interest in education reform, which is what drew me here to Hacker News. http://news.ycombinator.com/item?id=4728123 http://news.ycombinator.com/item?id=4728123
pflats 14y ago
They haven't figured out a damn thing regarding what makes a good teacher. At best, they've figured out how to find which teachers are good. That's not nearly as helpful. It's barely helpful. What happens when you find the bad teachers? Fire them and hope the fresh-faced college grads do a better job? So much of this debate on teachers is leans towards the carrot or the stick. Wanting to reward the best teachers because it makes you feel like you're helping is fine, but don't fool yourself into thinking it'll hugely improve instruction. Teachers aren't going to suddenly go, "Oh, wait, you'll pay me more if I'm a good teacher? I'm going to start trying!" (One or two might, but those outliers exist in any field, and are hardly the issue.) In every district I've taught, very little time is given for teachers of the same subject to collaborate, observe each other, and improve their own teaching. Every year, we have to get 20 hours of professional development. Tell me, are 6 minutes and 20 seconds of development per teaching day going to improve instruction?

77 points by eggspurt 14y ago

This work is valuable, but I personally would like to put more focus on improving teaching and learning - helping teachers do better, helping students do better, creating better learning environments and curricula. For example the budget of this study came out to about $15000 per teacher. What if instead we spent $5000 of that for professional development / training for the teacher, $5000 for a classroom set of chromebooks, and gave the teacher a $5000 raise. That alone would cause some significant improvements, I believe (not that that is ALL that is needed, and there are so many bigger issues that need to be tackled as someone else mentioned, such as poverty and inequity). But when we talk about "good" teaching and "bad" teaching, we are treating it like it is a simple craft that is mastered or not mastered. Teaching is like engineering, it involves design, and design is about making things better (see this prezi on learning design: http://prezi.com/b44jwdgvs8nl/olds-mooc-introduction/ http://prezi.com/b44jwdgvs8nl/olds-mooc-introduction/ )

jonknee 14y ago
Because this was a study, the money spent has little to do with the individual subjects. If pollster calls me for a a national opinion poll I would rather be paid 1/1000 of whatever the budget was than answer the questions, but obviously my participation is more helpful overall. The goal was to identify what actually works, which is something you need to know before improving the situation. Paying a bad teacher more will not make the education provided any better (despite whatever the teacher's union says).

dspeyer 14y ago

If I'm reading the linked page correctly, these metrics were verified by correlating them with individual student improvement on a variety of standardized tests and "more cognitively challenging assignments in math and english". This is better than pure circular logic, but until those metrics are validated, these are suspect. In particular, everything seems to test for shallow understanding.

murbard2 14y ago

If this becomes policy, it will most likely become another example of Goodhart's law. http://en.wikipedia.org/wiki/Goodharts_law http://en.wikipedia.org/wiki/Goodharts_law You can find a measure that correlates to performance, but once you use it to actually evaluate teacher, teacher will adapt and the measure will lost all relevance. Rely on test scores? Teach the test, not the material. Rely on student evaluations? Bribe them with easy material and good grades. Rely on principal evaluation? Get on its good side.

arocks 14y ago

Actually they have figured out how to evaluate teachers based on statistical models based on their repeatability of success. Finding a 'great' teacher is much more than finding the top rated teacher in my opinion. What qualities makes such a teacher good is a totally different matter. The latter was what I was hoping to find.

aaronharnly 14y ago

I, like many, hope the MET project yields fruit. However, see this post for a strong critique of one of the key claims of the MET study: http://garyrubinstein.teachforus.org/2013/01/09/the-50-million-dollar-lie/ http://garyrubinstein.teachforus.org/2013/01/09/the-50-milli... Namely, in the scatterplot correlating predicted effectiveness and actual effectiveness, they pooled teachers into 5% groups. This naturally reduces the variance, yielding a much straighter line of points than would an ordinary scatter plot. This is not, to my knowledge, an accepted or ordinary technique.

crusso 14y ago
Ugh, right from the get-go, Rubinstein throws away any attempt at statistical rigor by using an outlier example to try to prove something. The 5% groupings doesn't evaluate the metric. It just means that if they plan to use this actual study as a guideline, they should use the same technique in grouping teachers.
MarkMc 14y ago
I don't know enough about statistics to judge the strength of Mr Rubinstein's critique, but I find this a little odd: The axes on his graph is a 'raw score' between -1 and 1, but the axes on the MET study graph is 'standard deviations'. Is that significant?

nathan_long 14y ago

I can't say whether the findings are correct or not, but I like this: >> The teachers who seemed to be effective were, in fact, able to repeat those successes with different students in different years, the researchers found. Their students not only scored well on standardized exams but also were able to handle more complicated tests of their conceptual math knowledge and reading and writing abilities. To everyone who keeps saying "standardized tests are bad", please cut it out. A particular code benchmark may be useless, but benchmarking overall is good. Existing standardized tests may be bad, but in general, having some way to make objective measurements and adjust accordingly is the basis of all human improvement. And yes, we should reward good performance, if we can determine what good performance is. How else do you expect to get more of it? It is incredibly difficult to make incentives that actually do what you want rather than encourage gaming of the system, but that doesn't mean we shouldn't try. In short, let's stop debating whether we should measure and incentivize, and start debating how.

jgj 14y ago
I'm not sure that the argument is even that existing standardized tests are bad so much as that they are given too much priority. They should not be the only or primary means by which teachers, students, districts, etc are benchmarked (and in turn, rewarded or disciplined). "Teaching the test" is the problem, not necessarily the tests themselves.
UnoriginalGuy 14y ago
I grew up in the UK in a school where the emphasis of our entire education was "the test." Teachers entirely forgot how to teach a subject and our whole time was taken up with what essentially amounted to "test prep." I did pass their tests, but did I get a good education? Frankly, no, I don't think I did. I mean statistically I got a great education, but it seems we have forgotten why we even educate to begin with... Do we educate so people look smart on some random league table? Or do we educate so people are actually more well rounded, have a better grasp of the world around them, and do better as adults?
- crusso 14y ago

axelfreeman 14y ago

"We’ve figured out what makes a good teacher" They've figured out that a "good" teacher can repeat the process of success (whatever this is), there is no information how he made it. The headline is slightly misleading.

jonknee 14y ago
The good news is they have video of good teachers (and bad) at their craft for years. It will take more analysis, but they have all the data at the ready.

UnoriginalGuy 14y ago

Using test scores in order to judge the quality of a teacher is like using brush strokes in order to measure the quality of a painter.

crusso 14y ago
So where part of the study showed that with randomized redistributions of students, certain teachers were able reproduce higher test scores -- your countering evidence shows otherwise. Great. Where's your evidence? My own evidence (experience going through school and generalization of how ability works in most professions) tells me that the good teachers in my life helped me to learn the information better and more deeply. My memories of those classrooms are of the subject matter and my desire to achieve in that teacher's environment. The crappy ones were noticeably so in that my memories of those classrooms was of misbehaving with my friends, having no desire to help the teacher teach me, and cramming to learn material for tests because I hadn't absorbed anything beforehand.
- sethammons 14y ago
  I just realized that I keep replying to you, crusso (lol). Cheers. I'm more interested in the demographics of the student population than the 'random distribution (1)' of students in that population to different teachers. What I mean by this is that I would expect 'good' teachers to be able to demonstrate their abilities with with relatively homogeneous groupings of students where those students are from relatively decent neighborhoods. If, however, this study included teachers in inner city schools with huge variance in the 'quality' of student that a teacher would receive at often vastly different levels of understanding in the teacher's discipline area, then I think the results would be interesting insofar as to allow others to dig into what makes these teachers effective. 1: If we are looking at a teacher who teaches a course with a pre-requisite, the grouping of students becomes more similar (ie, a trigonometry teacher is likely to have students both motivated to do well and to have passed the requisite lower level math classes, and their success is less surprising to me).
  - crusso 14y ago

cbsmith 14y ago

I'll have to look at the prior reports, but this 3rd report is somewhat disappointing in terms of its statistical rigor. They do provide weightings based on optimal predictions for maximum accuracy with predictions for state tests scores, although those weights appear to assume a simple linear correlation. Even then, the data suggests that in class observations are almost a complete waste of time (single digit weightings), and that student surveys decline in value as the kids mature (no shocker: with younger kids, how much they like the teacher has a much more profound impact on their success at learning). In all cases the kids seem to be a more reliable indicator than the observations of professionals! This may just be an indication that assessments based on more frequent observations are more accurate, but the only way we can really get a sense of that is if they also had peer observations from team teaching environments... which are completely absent. That's not the worst of it though. The worst is the rest of the data. It is based on some fairly arbitrary presupposed models for combining these different assessments. No breakdown of the individual components of these assessments (other than "English" & "Math") and no attempt to discern a model from first principles. There are some other factors as well. Aside from some outliers, the accuracy of predictions seems kind of suspect. Most of the outcomes are within +/-0.05 standard deviations, and a majority of them are within what looks like +/-0.025 standard deviations. These are pretty minuscule variances (0.25 was supposed to represent a year of schooling, so 0.025 represents maybe 15 school days worth of progress at best), and it is not hard to see that without some of those outliers, the regression line doesn't look so pretty (particularly for English). I also didn't see any metrics on the variance of the data within a classroom, which I suspect makes any predictive value look limited at best. Ugh. I really hoped that Gates would make sure there was more statistical rigor with an analysis like this. :-( I hate to say it, but the conclusions of the study are not well supported by at least what is in the final research report.

wheaties 14y ago
Yes, this is also the same foundation that spent 1 billion to make schools smaller, because they didn't do simple statistical analysis to find that smaller schools showed much larger variability. Now they're coming back to say "oops, we were wrong." http://marginalrevolution.com/marginalrevolution/2010/09/the-small-schools-myth.html http://marginalrevolution.com/marginalrevolution/2010/09/the...

madaxe 14y ago

"Test scores" Brilliant. Well, yes, that is the best way to measure a teacher's efficacy, if your goal is to teach students to jump through hoops and pass standardised tests rather than to think. The best teacher I ever had was my grandmother - she taught me the most important lessons I'd ever learn, such as: "Observe the world around you." "Reach your own conclusions." "What was fact yesterday may be fiction tomorrow." Tests are a distraction from learning.

coob 14y ago
Did you read the article? It didn't use just standardised test scores.
- madaxe 14y ago
  Of course I did, but standardised tests, particularly in the US, do little other than teach people how to game multiple choice, and pass tests. Life isn't about passing tests, it's about learning.
crusso 14y ago
I don't really understand the visceral reaction against testing in this HN thread. You're tested throughout your working life. Almost every job interview is a test. Giving a customer presentation is a test. Finding bugs in your software for a release date is a test. Memorizing that big Cheesecake Factory menu so you can wait tables is a test. Getting your General Contractor's license is a test. Getting certified as a Pilot requires passing a test. There should be more to education besides testing, but being tested is a part of life and it's easy to measure success - that combination means that it's a useful tool for education.
- Mvandenbergh 14y ago
  Job interviews are a good example of a kind of testing that we know doesn't work all that well but we keep around for lack of a better alternative. The others are not good examples because they are useful in-and-of themselves. Giving good presentations and finding bugs are useful because they are useful, not because they predict something else. Pilot's licenses, like driver's licenses are based on tests where you more or less do the thing that you're being tested on. The test and the thing that it's supposed to measure are very similar and we can therefore use it to predict performance. In cases like that "teaching to the test" doesn't exist because practicing for the activity and for the test is the same thing. I'm not denying that educational testing is useful to quantify progress but such tests are often different enough from the underlying activity they're intended to measure that effort spent preparing for the test has little other purpose.

gnosis 14y ago

There are many important factors which teachers (no matter how good they are) can't control: 1. How wealthy, poor, educated, or uneducated a child's parents are. 2. How many resources the school and teacher are given. 3. Whether a child's parents are abusing them at home. 4. Whether a child is witnessing violence, hopelessness, despair in their neighborhood. 5. The attitude of a child's family and friends towards education. 6. Whether a child's parents are active in their child's education and in a PTA. Study after study has shown that these factors greatly influence a child's attention and attitude to school work, their attendance, and ultimately their performance at school. No child's performance on a standardized test, and no evaluation of a teacher's performance is going to pick up on the above factors, yet they can critically impact how well the child ultimately learns and what the child learns. Standardized testing of a child that is unfortunate enough to live in a situation where the above factors are against them will likely notice a poorly performing student, one who gradually does worse and worse on the tests until they finally flunk out. Then, instead of getting extra aid, the funding to the school that has a lot of such children will be cut (thanks to programs like No Child Left Behind), and their teachers penalized. These factors, which are strongly associated with poverty, are things a given teacher is in no position to influence, but which society as a whole could address. If it has the will.

Mvandenbergh 14y ago
That's why they do value-added testing rather than just evaluating teachers based on the score of their pupils. Obviously there are confounding factors, but if a teacher can make a statistically significant difference to the the test scores - as measured by before and after testing then that is valuable to know.
Symmetry 14y ago
So, if a teacher has as little effect on a child's performance as you seem to be implying should we then reduce teacher pay, accept the lower quality applicants we would get since it doesn't matter, and spend the extra money on anti-poverty measures?
crusso 14y ago
And at my job, there are many factors out of my control. I don't control the home life of my engineers. I don't control my customers' decision making processes. I don't control my competitors' actions in the market. But yet I'm judged on the small slice of that overall pie that I can control. What this study indicates is that there are teachers who can reproducibly create success levels given the control of the situation that they do have. How is it any different than the rest of the working world where you can control some things and you can't control others yet your job is to do the best you can with what you can control?

the_bear 14y ago

It's common for people who received good educations to say that test scores aren't an acceptable indication of learning. It's probably true that if you already have an acceptable level of education, test scores shouldn't be too important, but it seems like the people criticizing test scores haven't experienced what education is like for the children being "left behind." In middle school, the last three weeks of each semester were spent watching movies (the same two or three movies over and over). In high school, I was taught that the big bang theory was when the continents divided. I was taught that the most direct route between two points on the globe is through the north pole. My Spanish and woodworking teachers taught math because they knew that the math teachers weren't doing it. When you're talking about schools like the ones I went to (which weren't nearly as bad as many other schools in America) I think that test scores are a decent indicator of the education a student is receiving. You're measuring things like "can this kid read English" and "can they multiply two numbers together." Standardized tests are certainly capable of measuring things like that. Testing probably breaks down for measuring intelligence/education as the level of education improves, but I encourage everyone to consider just how dysfunctional many schools are before ignoring testing entirely.

roguecoder 14y ago
Except that even those things don't predict later success. Any kid with a family that can afford books will score higher on "can this kid read English". Any kid who's been drilled can learn to multiply two numbers together. Without cognitive skills behind them, though, those are not useful abilities. The best predictors of future success are self-control, emotional intelligence and a grasp of functional algebra (which I believe, but don't have evidence to support, is because of the importance of computational and systems-level thinking.) Only one of those, self-control, is even indirectly measured by current standardized tests.
- ims 14y ago
  > Any kid with a family that can afford books will score higher on "can this kid read English". True, but there are also many families that can afford books but don't buy them, instead spending money on other things. Not having books in the house has been shown to correlate with decreased academic success.[1] I'm not saying this isn't a wealth issue, but that probably isn't the only dimension. [1] http://www.sciencedirect.com/science/article/pii/S0276562410000090 http://www.sciencedirect.com/science/article/pii/S0276562410...

brudgers 14y ago

The study identifies teachers who are effective against a particular metric. Most identified are likely members of the subset of good teachers. Yet, the three R's as measured by standardized test outcomes are not the only measure of effective education, just one that is cheap to measure. Dewey believed that the goal of education should be to produce good citizens. Systems with metrics based upon the model of student as consumer are more likely to produce outcomes biased toward producing consumers. The entire set of good teachers includes those who inspire a lifetime of learning. It includes those who teach kindness, sportsmanship, and good humor. It includes those who teach subjects for which there are few students and in which standardized tests make little sense such as art and shop. Maybe the study can point us in the right direction. But few districts will bring outside evaluators from Cambridge's famed institutions. Few districts will fund ETS analysis or purchase their tools. What we are likely to get are processes localized to the political reality on the ground.

robdoherty2 14y ago

I believe that studies like this are well-intentioned in that those conducting the research are truly trying to improve the state of education in the US. However, the result of their work I'm afraid is the 'quantification' of teaching as a profession.

jnazario 14y ago

kudos to the WaPo for giving a link to the study. too often, many publications discuss a study but never link to it, which then makes it harder to go and review it for yourself.

blueprint 14y ago

The title of this article, and perhaps the researchers' conclusions from their data, is wrong. They didn't find the criteria of a great teacher (that is, what makes a good teacher). They just found some correlations among the _qualities_ of good teachers. It's the total opposite meaning. They found results of good teachers, not causes.

loucal 14y ago

Is this the same gates foundation that I saw on 60 minutes pumping GMO seeds and fossil fuel fertilizer to third world countries to "help" them out of their food crisis? In fairness, I think I heard they realized something like "what works in america doesn't work everywhere" (bit of a cop out to save face) and moved toward more sustainable organic methods, but I'm not completely sure. All I know is, if its that hard for them to figure out the most practical way to grow a plant, we should probably take their advice on growing the minds of our children with a grain of salt.

atomical 14y ago
There's nothing wrong with using GMO seeds to increase yield.
- loucal 14y ago
  It is opinion either way but in my opinion not only isn't it necessary, but it clearly threatens global biodiversity. Proper sustainable farming techniques produce an abundance and a variety of food, so the extra 1% increase on a huge monocropped field isn't even something that makes sense for third world communities. On the biodiversity thing, viewing your website leads me to believe you appreciate nature, so I'm just scratching my head that you haven't made the connection. Ohh well, downvote me all you want buddy. You are entitled to your opinion also.
  - atomical 14y ago
    It's much more than 1%. I haven't down voted you. Liking nature really has nothing to do with anything else.

MaysonL 14y ago

Evaluating teachers is all well and good, but the key to improving our educational system is not that. What is needed is improving teachers. See "Building a Better Teacher"[0] [0]http://www.nytimes.com/2010/03/07/magazine/07Teachers-t.html?ref=magazine&pagewanted=all&_r=0 http://www.nytimes.com/2010/03/07/magazine/07Teachers-t.html...

tokenadult 14y ago

The article kindly submitted here links to the project homepage http://www.metproject.org/ http://www.metproject.org/ at which the latest project report can be found. I've got some reading to do before I think about what else to add to the discussion already underway here. I get the impression that one observation that has motivated much of this research is how persons educated overseas have fared in United States society after immigrating here in early adulthood. My own experience living in another country (twice, in two different three-year stays) does much to prompt my interest in education reform, which is what drew me here to Hacker News. http://news.ycombinator.com/item?id=4728123 http://news.ycombinator.com/item?id=4728123

pflats 14y ago

They haven't figured out a damn thing regarding what makes a good teacher. At best, they've figured out how to find which teachers are good. That's not nearly as helpful. It's barely helpful. What happens when you find the bad teachers? Fire them and hope the fresh-faced college grads do a better job? So much of this debate on teachers is leans towards the carrot or the stick. Wanting to reward the best teachers because it makes you feel like you're helping is fine, but don't fool yourself into thinking it'll hugely improve instruction. Teachers aren't going to suddenly go, "Oh, wait, you'll pay me more if I'm a good teacher? I'm going to start trying!" (One or two might, but those outliers exist in any field, and are hardly the issue.) In every district I've taught, very little time is given for teachers of the same subject to collaborate, observe each other, and improve their own teaching. Every year, we have to get 20 hours of professional development. Tell me, are 6 minutes and 20 seconds of development per teaching day going to improve instruction?