14 ms·

Teacher Effects on Student Achievement and Height: A Cautionary Tale

237 points by blfr 7y ago

deleted 7y ago
[deleted]
thepete2 7y ago
What if bad students get hit on the head thus decreasing their height?
- dang 7y ago
  Please don't do this here.
joshvm 7y ago
This would have been a perfect opportunity to cite the paper about the efficacy of parachutes: https://www.bmj.com/content/363/bmj.k5094 https://www.bmj.com/content/363/bmj.k5094
- jotakami 7y ago
  Too slow to make the connection here... can you explain your reasoning?
  - jk563 7y ago
    The "Teacher Effects" paper is demonstrating that the value add to student achievment is likely noise and not representative of the teachers doing a better job. The "Parachutes" paper is demonstrating that parachutes make no difference when used out of a stationary grounded plane. I think the point is that both papers are demonstrating that the numbers alone don't tell the full story.
    - jotakami 7y ago
      The parachute paper says this, which I think is their “punchline”: When beliefs regarding the effectiveness of an intervention exist in the community, randomized trials might selectively enroll individuals with a lower perceived likelihood of benefit, thus diminishing the applicability of the results to clinical practice. So... was there some kind of selective enrollment in the teacher study? I don’t think that’s the issue here.
      - iguy 7y ago
        Sure, not the same mechanism (selective enrollment). But the papers seem parallel in that they try to demonstrate something we know to be crazy, using steps that are accepted when studying things we don't know.
  - goto11 7y ago
    It seems to be a contrived joke: They tested people jumping with and without parachutes from a small airplane situated on the ground.
    - pmyteh 7y ago
      It's a follow-up to a previous article bemoaning the lack of a randomised controlled trial of parachutes and arguing that they should not therefore be accepted as a valid 'treatment' for falling out of an aeroplane (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC300808/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC300808/). Both this article and its predecessor were published in the Christmas issue of the BMJ, where funny papers with a semi-serious point (in these cases, the ongoing debates in medicine about the superiority of RCTs over other forms of clinical evidence) are common. My favourite of this genre is "Does Peppa Pig encourage inappropriate use of primary care resources?" (https://doi.org/10.1136/bmj.j5397 https://doi.org/10.1136/bmj.j5397) critiquing the clinical practice of Dr Brown Bear, a fictional cartoon GP.
      - CrazyStat 7y ago
        My favorite BMJ Christmas article is a study of self-reported virgin mothers [1], but I might be biased because I know one of the authors. [1] https://www.bmj.com/content/347/bmj.f7102 https://www.bmj.com/content/347/bmj.f7102
    - Ma8ee 7y ago
      Randomized studies of new treatments often exclude the sickest patients that could have benefitet most from the treatment. In particular, there won't be any really seriously ill patients in the control group because they will most likely die without treatment. That skews the results of these studies. This is what they try to illustrate with the jumping out of plane study. The result won't be reliable if you don't have a control group that actually might have benefit from the treatment. Cancer patients (jumpers), will or will not get a new type of chemotherapy (parachute). But since it is known that the sickest (hightest jumping hight) certainly will die without treatment (parachute), they decide to only test the treatment on patients with only early stage and benign cancer (close to the ground).
2stop 7y ago
Research done more than 20 years ago (and repeated and confirmed) already found that teachers have very little effect on educational outcomes. The biggest correlative factor is socioeconomic status and education level of the parents. Everything else is hit or miss But hey, let’s publish a pointless paper... because academia.
- dooglius 7y ago
  According to the paper, many states are still using VAM so it bears repeating.
- sethammons 7y ago
  I think this is an important paper. Even or especially if it confirms other studies. When I was a teacher in inner city San Bernardino, we were held accountable for student achievement ignoring student ability, socioeconomic situation, home life, or their capacity to disrupt the class. In large part, this is because of studies that say a teacher has outsized effect on student test scores. Now, I think a special kind of person can connect better with some kids and help them achieve. I think that is not something you can mass produce. With more studies showing teachers are not as critical as schools want to believe, maybe we can start focusing on other metrics to gauge suçcess.
  - ckcheng 7y ago
    > With more studies showing teachers are not as critical as schools want to believe, maybe we can start focusing on other metrics to gauge suçcess. Like cost of hiring? Seriously, if teachers are not as critical, why not hire less or pay less salary... any half decent person in the classroom will do. I think the study is about right that teachers are not as critical as they'd like to believe. I just don't know what to think of the implications...
    - sethammons 7y ago
      I still think a bad teacher is absolutely terrible and that teaching is not easy. I don't look at teacher salaries and think they are too high at all. I think we are bad at measuring what makes a good teacher. I just know it is not based on standardized test scores. When I was a teacher, I had zero power to remove a disruptive or inattentive student from my class. I had 9th graders who could not deal with negative numbers being expected to do well on state Algebra tests that are of arguable quality. Even if they made substantial progress, they would not be to grade level. I can still remember one student who after nearly a year of not being bothered to pay any shred of attention in class, during the final review, paid enough attention during the last steps of solving an equation: Me: Alright, and combining like terms, what is 10x - 18x? Student, suddenly paying attention: But wait, you can't subtract a bigger number from a smaller number! I can't stop the class on nearly the last week of 9th grade algebra to help this student understand negative numbers. And they are not interested in coming in to get help outside of normal class. Parents were not interested in making sure this kid got through school. Not so fun fact: at this school, less than 4% would go onto any post-secondary education. Of them, about 2% would go on to finish a degree. Something like a 90% transitory rate (meaning that most students who started 9th grade at this school would transfer to a different school before graduation, if they made it that far). I could go on, but there was an underlying cultural issue that did not value education. That trend is hard to buck: I had third generation gang members, kids raising their siblings because parents (often a single parent) were working multiple jobs, kids dealing with daily violence, one kid was stabbed to death right off of campus. The majority of these kids and their families have no experience seeing what an education can do for their prospects. Add onto that a tough employment market (inner city San Bernardio!), and those who did have siblings who did get a degree, they often couldn't find a job. All this to say, I don't think lowering salary of teachers would help. Maybe lowering the cost of administrators would help. At this same time, there were more administrative personnel in the district than teachers, something I could never understand.
pjc50 7y ago
It's not totally beyond the realm of plausibility for teachers to have long-term effects on height, given the effect of childhood nutrition on height. Properly feeding children can have such a positive effect on their academic achievement that some teachers end up doing it out of their own pocket. Certainly the best way to raise the worst-performing children is "fixing" whatever disaster family situation they are dealing with at home.
- deleted 7y ago
  [deleted]
- lonelappde 7y ago
  Teachers are not feeding kids in statistically significant volumes. Schools are, and there might be correlations.
  - ChrisLomont 7y ago
    >Teachers are not feeding kids in statistically significant volumes. Likely untrue, given a few moments search. For example, "On average, teachers spend $37 a month and principals spend $59 a month for food for their students." [1]. There's lots more data on teachers feeding kids in perhaps causal amounts. [1] http://neatoday.org/2013/09/10/report-teachers-spend-37-every-month-to-help-feed-hungry-students/ http://neatoday.org/2013/09/10/report-teachers-spend-37-ever...
    - dooglius 7y ago
      How many students do teachers have on average? $37 per month definitely does not seem significant enough to explain the effect found.
      - ChrisLomont 7y ago
        Avg public school teacher:student ratio is 16:1, avg school meal costs $2.48. This may be a non-negligible effect. Given the strong correlation between some breakfast vs none, and school performance, it would not take much to affect the outcomes by feeding hungry kids little snacks.
jotakami 7y ago
My son started kindergarten this year, and unfortunately we are zoned into one of the worst schools in the district. He’s a bright kid, already knows everything they would teach him, so we weren’t concerned about academics, but let me tell you—his behavior at home, and especially his language use, has taken a really troubling turn. He is obviously picking up the behavior of some of the children he goes to school with. I read a book not too long ago entitled “The Nurture Assumption,” by Judith Harris. It influenced me more than anything I’ve read as a parent, because it is so obviously true. Her thesis is that everyone assumes that adults have a significant impact on a child’s future, yet there is basically zero evidence to support such an assumption. Peer groups are the dominant influence on a child’s personality and development, full stop. This is probably hard for educators to accept because there’s really nothing that can be done about it from their perspective.
- rimliu 7y ago
  So loners are not influenced by anything?
- Ididntdothis 7y ago
  I don’t understand why people often want to reduce something to only one factor that explains everything . When I look back at my childhood I was influenced positively and negatively by my parents, siblings, teachers, my peers, neighbors and a lot of other people. They all had an influence at times that deeply impacted my life.
  - lallysingh 7y ago
    For planning and resource application purposes, you can only sell effort on one factor at a time to non-specialists (policy makers, parents, the public)
    - Ididntdothis 7y ago
      That's how you get terrible planning. Maybe we should just accept that things are not that simple and there are a lot of things that influence complex systems.
bhouston 7y ago
This is also the tale relevant to most A/B testing.
jefftk 7y ago
Summary: schools are rewarding teachers based on "value-added measures" that try to measure the effect of the teacher on the student's educational progress. The paper shows that these models are mostly picking up unrelated differences between student populations, by showing that the same approach would reward teachers for having taller students.
- Accujack 7y ago
  Oddly, most school basketball programs DO get rewarded for having taller students.
- klipt 7y ago
  Our society has decided that taller is better, so naturally the best (tallest) students should get the best (tallest) teachers!
  - Spooky23 7y ago
    My son’s teacher is 4’11”. I’m filing a complaint immediately.
- wtvanhest 7y ago
  Could be a landmark paper. At a minimum, it shows that all evaluation methods should completely negate teacher's impact on height. It seriously calls in to question a ton of research that I would have assumed to have been good. Excellent work by those that wrote the paper.
  - datashow 7y ago
    It appears to be an important research study, but in the sense of implication to educational policy, not so much. The reason is, for one, the use of value-added measurement does not have a significant role as many would think. There is not a signle state in the U.S. use VAM as a single measure of teacher performance, nor a determinant factor in teacher firing or promotion. It is always mixed with other quantitative and / or qualitative measures (which usually are no better than VAM). The second reason is, the research community has not find a better replacement yet. We always know we cannot directly use student achievement, now we know we cannot trust a more complicated measure such as VAM which takes into account of student history and other factors.
omarhaneef 7y ago
I think the message to parents is loud and clear: send your kids to the teacher that will make them tallest.
- _benj 7y ago
  You have made me laugh dear sir or ma'am! :P
nicoburns 7y ago
I'd strongly question our ability to measure educational outcomes. If they're basing that on standardised testing, then I'd argue that pretty much invalidates any conclusions of the research.
- dissidents 7y ago
  Do you think it's inherently impossible to measure educational outcomes or just difficult with the tools that we today?
  - nicoburns 7y ago
    I think it's very difficult to come up with accurate objective measurement methods (nobody's done it well yet). The best assessments tend to be subjective teacher-based assessments (for similar reasons as to why hiring tends to be done by subjective personal assessment - it's hard if not impossible to define the criteria for "good" upfront. Someone may approach things in a completely different way that nobody had thought to measure). And this is before you get into the biases of the people drawing up the assessment criteria etc.
petrogradphilos 7y ago
> 1 Introduction > The increased availability of data linking students to teachers has made it possible to estimate the contribution teachers make to student achievement. There was some data available before (or the sentence would not have used the word "increased"). Why wasn't it possible to estimate with that? > By nearly all accounts, this contribution is large. It goes on to talk about what "large" means: > Estimates of the impact of a one standard deviation (σ) increase in teacher “value-added” on math and reading achievement typically range from 0.10 to 0.30σ, which suggest that a student assigned to a more effective teacher will experience nearly a year's more learning than a student assigned to an less effective teacher (Hanushek & Rivkin 2010;...). (Typo: "an less effective" should be "a less effective".) A "range from 0.10 to 0.30σ" doesn't make sense. A Greek lowercase sigma (σ) is used to represent one standard deviation, but the sigma is used only on the upper end of the range. Should it have been from 0.10σ to 0.30σ? And how are they measuring the impact on achievement of an increase in teacher "value-added", anyway? It says that estimates of the impact "typically range from 0.10 to 0.30σ", but it doesn't say what units those figures are in. The sentence goes on to say that those unit-less estimates "suggest" that "a student assigned to a more effective teacher will experience nearly a year's more learning than a student assigned to an less effective teacher". Over what time period? That is, how long does a student have to study under a "more effective teacher" to get "a year's more learning"? 1 week? 12 years? It doesn't say. And finally, how do those unit-less estimates "suggest" an impact measured in learning time? It doesn't say.
- 6gvONxR4sf7o 7y ago
  You are being absurdly pedantic. For example >A "range from 0.10 to 0.30σ" doesn't make sense. Out loud, you might read it as "oh point one to oh point three sigma." And it's fine. Ever written a phrase like "it's very expensive, costing somewhere around $3-4M" with one "$" and one "M" but two numbers?
- Mathnerd314 7y ago
  NYC data goes back to 1999, but only for some basic things like test scores. They probably don't have the consolidated links between students and teachers outside of the years they analyzed, due to differing data formats and general laziness. It's a terrible paper in general though, for example it references Bitler et al. 2015 but never provides a full citation.
brohee 7y ago
Can't econonomists learn LaTeX? The typesetting is so horrible its putting you off from reading it...
deleted 7y ago
[deleted]
deleted 7y ago
[deleted]
snarf21 7y ago
"When a measure becomes a target, it ceases to be a good measure." - Goodhart's Law
jeffdavis 7y ago
If somebody pays for something, they want to see the results. If the effects are local, you see with your eyes and you hear things second hand; a kind of soft feedback. If the effects are far away, you probably want to see a hard number, because you don't get the same kind of soft feedback. Is the number (teacher score) useful? Maybe not. But you can't simply take away a number, resulting in no feedback. You have to either come up with a new number that is useful, or localize it enough that people can see the results. Otherwise people won't want to pay. That's why these weird bureaucratic numbers usually come from state or federal requirements. People see the big pile of money going somewhere and they want "accountability" because they can't see the results for themselves. Let's stop shuffling around so much money, laundering it through state and federal agencies. The money, if it comes back, always comes with strings attached. This goes for schools, highway funds, and many other things.
jedberg 7y ago
The problem with using height is that there is already evidence that taller people are more successful due to hidden biases. So perhaps the teacher measures were still valid because all the tall people were more successful because of other biases in the system? I'm mostly just being cheeky here. To be clear, I still think it's nearly impossible to measure how good a teacher is based on student performance, since the teacher is only one part of the equation, the parents (and the student's home life) being a much bigger part of the equation. The teacher can only do so much if the child comes home to parents that don't feed them and tell them they are worthless.
- telotortium 7y ago
  Perhaps, but the point is that the value-added statistics that have been used (in trials, I don't know if this was ever into a formal metric with consequence) to measure how much students' grades or test scores are improved by a particular teacher's performance show almost as large of a supposed effect of the teacher on height, not height on grades.
  - jedberg 7y ago
    The study says that the measure of teacher performance is correlated to student height. The measure of teacher performance is supposed to be correlated to student success. What I'm saying is that if student success is already correlated to student height, then anything that measures student success will also measure student height.
    - JamesBarney 7y ago
      While what you say so true if they're ere just measuring student height/success. But they're not measuring student height/success they're measuring the effect of teachers on student success/height over time. And what you said isn't true of that measure.
      - jedberg 7y ago
        Student success over time is correlated to student height. It would be impossible to measure one without measuring the other.
MarcScott 7y ago
I worked as a teacher in, state funded, education for 15 years. Value added measurements of student achievement will always fail. In affluent areas, where student achievement is already high, schools can afford to hire the best teachers, but they will realistically only be able to boost achievement by a small fraction, given the child's prior attainment. In deprived areas, where student achievement is lower, schools can only afford to hire less qualified teachers, and the boost to achievement will be a small fraction of previous attainment. Either way, the measured effect that the teacher has will be marginal, while common standards and common testing are the only measured outcomes. This does not equate to the teachers not being invested in student outcomes, or trying their best for their students. Value added measures were only introduced as a form of target setting, for teacher performance. I'd encourage everyone to watch Adam Curtis' The Trap[1] to see the effects of target setting. Teachers in early years education set a target, then primary school teachers need to show value added so game the system to show improvement. This trickles up the system, and the net result is unrealistic targets for almost every child as they exit formal education. [1]https://en.wikipedia.org/wiki/The_Trap_(TV_series) https://en.wikipedia.org/wiki/The_Trap_(TV_series)
- jim-greer 7y ago
  What do you think is the best way to evaluate teachers, if any?
  - germinalphrase 7y ago
    I teach at the high school level. I don’t think we have figured it out yet; however, my personal belief is that it would require the crevasse between the high-volume/low-stakes instruction-focused assessment (which cannot be “trusted“ generally) with standardized assessment (which can be “trusted”)to be bridged in such a manner that the daily assessment that takes place all across the country, every single day, can be viewed as legitimate measures of student capability. Without being able to accurately measure student ability day to day, we will not be able to measure teacher effectiveness. I would love to see a toolset like this come into being, but I have no ability to develop such tools.
sadness2 7y ago
It seems to be universal that basing people's career progression on metrics is harmful. People try to do this with engineers as a substitute for understanding what's happening on the ground and having an actual relationship. At best, metrics can raise a flag for you to ask empowering questions.
meekstro 7y ago
I know a brilliant junior school teacher teaching years 1 to 3. He constantly configures his classroom to maximise learning continuity for the entire class. It is a massive challenge balancing out the problems students face physically or outside the classroom. Teaching has become the easy part of teaching. The problem with student learning is that is affected by three things. Emotional Health Physical Health Teaching Quality to build an educated adult you need to optimise all three variables. The problem with rewarding teachers is that their students aren't objectively assessed on all three dimensions. In every district in the world there should be three person educational assessment teams that consist of an emotional health assessor, a physical health assessor and an academic progress assessor that work as a team to objectively assess the students and work with the teacher and support agencies to eliminate learning barriers. Any students that need help should get referred to the correct service or program as quickly as possible. The teacher should focus purely on teaching and integrating support agencies. Teachers deserve to be rewarded for their teaching ability in an objective way. As it happens there is a lot of unnecessary subjectivity and people blame teachers for not being social workers, counsellors and doctors. The education of every child in a class deteriorates as these three dimensions become less homogenous because the maximised learning continuity of the group depends on a limited resource. It is teachers who can creatively balance them while engagingly delivering the curriculum that are creating unrewarded alpha in the educational space. Paying $250,000 a year in salary to a group of 3 specialist objective assessors with the data linked to the health and social services would get a better return on investment than any other expenditure a government could make in lifting the health, education and future earning potential of children. If a child isn't learning there is always a reason. The faster you address that reason the more value you create. Sometimes it's the teacher, sometimes it isn't but for the child's sake we should definitely diagnose the reason.
deleted 7y ago
[deleted]
neallester 7y ago
For children who are still growing physically and mentally, both height and intellectual development are correlated with age, aren't they?. In an analysis where you are treating children who actually have a range of ages as a single age (i.e. "5th graders") finding some correlation between height and intellectual development isn't that surprising. It is just an expression of the underlying correlation between their age and intellectual development.
- xyzzyz 7y ago
  Height is generally correlated with IQ, though not very strongly (r = 0.2), and IQ is very strongly correlated with education success. It is also known that this correlation is not spurious: half of it is pleiotropy, which is, some of the genes that cause taller height also cause higher intelligence, and half of it is due to assortative mating, meaning taller people tend to have children with other taller people, so if pleiotropy causes taller people to be more intelligent, the tendency to marry taller people additionally amplifies the correlation. Keep in mind though that the correlation, while very highly statistically significant and replicated over many data sets, is not very strong: at r = 0.2, the r^2 is 0.04, so only 4% of variance in intelligence is explained by height. Thus, it is not clear whether the height-IQ correlation plays significant role as a confounder in the study above.
  - iguy 7y ago
    But the numbers you describe are for adult height, right? Wouldn't you expect a much stronger effect if your sample was a mix of (say) 4-6th-grade kids. I think that's what GP is saying.
    - Beldin 7y ago
      Kids do not grow in lock-step. I.e., kids who will have exactly the same height as adults will have (probably considerably) different heights at the same age while growing up. Given that, i doubt height measured at any one (age)point during childhood strongly correlates with iq. You'd need sufficient data to be able to reliably predict height as an adult for that, would be my guess.
mcguire 7y ago
Hypothesis: Teachers, individually, don't have statistically measurable effects. Students in a given cohort are mostly identical, and are mostly exposed to identical environments, such that influences on them average to some mean. As a result, minor influences, as long as they act reasonably consistently, are elevated to statistical significance.
- datashow 7y ago
  On average, it would be true, meaning most teachers do not have significantly different effect on most students. However, there are always particularly good and bad teachers.
drited 7y ago
Ruling out teacher impact on height without testing is perhaps not strictly scientific? We really don't know what is true unless we test it. I know it sounds ridiculous but it's not beyond the realms of possibility that teacher opinion of student capabilities influences social status which in turn influences release of growth hormones which influence height. Aren't there other species which grow when they rise to the top of the pack as growth hormones are released to assist in maintaining that status? Not saying it's likely but if left untested we cannot rule that out. There have been plenty of surprises in science when we went and actually tested what we previously presumed to be implausible.
- youzicha 7y ago
  Or perhaps good teachers are also better at getting social services to poor students, so they can get food stamps, eat better, and grow talller? https://xhxhxhx.tumblr.com/post/189336550187/teacher-effects-on-student-achievement-and-height https://xhxhxhx.tumblr.com/post/189336550187/teacher-effects...
droithomme 7y ago
There's a significant positive correlation between height and IQ: https://en.wikipedia.org/wiki/Height_and_intelligence https://en.wikipedia.org/wiki/Height_and_intelligence There's also a correlation between IQ and academic achievement. So, if we did decide for some (undoubtedly bad) reason to reward teachers for happening to have more than average tall students it would be comparable to rewarding them for having more than average high achieving students.
paggle 7y ago
Anyone who has done A/B testing knows that you need tens of thousands of data points to pick up any causal relationship unless it’s a huge effect like the retention impact of telling your customers to lick a donkey’s balls. No way you could pick it up for a teacher with 30 kids a year except over that teachers entire career.

237 points by blfr 7y ago

deleted 7y ago

[deleted]

thepete2 7y ago

What if bad students get hit on the head thus decreasing their height?

dang 7y ago
Please don't do this here.

joshvm 7y ago

This would have been a perfect opportunity to cite the paper about the efficacy of parachutes: https://www.bmj.com/content/363/bmj.k5094 https://www.bmj.com/content/363/bmj.k5094

jotakami 7y ago
Too slow to make the connection here... can you explain your reasoning?
- jk563 7y ago
  The "Teacher Effects" paper is demonstrating that the value add to student achievment is likely noise and not representative of the teachers doing a better job. The "Parachutes" paper is demonstrating that parachutes make no difference when used out of a stationary grounded plane. I think the point is that both papers are demonstrating that the numbers alone don't tell the full story.
  - jotakami 7y ago
    The parachute paper says this, which I think is their “punchline”: When beliefs regarding the effectiveness of an intervention exist in the community, randomized trials might selectively enroll individuals with a lower perceived likelihood of benefit, thus diminishing the applicability of the results to clinical practice. So... was there some kind of selective enrollment in the teacher study? I don’t think that’s the issue here.
    - iguy 7y ago
      Sure, not the same mechanism (selective enrollment). But the papers seem parallel in that they try to demonstrate something we know to be crazy, using steps that are accepted when studying things we don't know.
- goto11 7y ago
  It seems to be a contrived joke: They tested people jumping with and without parachutes from a small airplane situated on the ground.
  - pmyteh 7y ago
    It's a follow-up to a previous article bemoaning the lack of a randomised controlled trial of parachutes and arguing that they should not therefore be accepted as a valid 'treatment' for falling out of an aeroplane (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC300808/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC300808/). Both this article and its predecessor were published in the Christmas issue of the BMJ, where funny papers with a semi-serious point (in these cases, the ongoing debates in medicine about the superiority of RCTs over other forms of clinical evidence) are common. My favourite of this genre is "Does Peppa Pig encourage inappropriate use of primary care resources?" (https://doi.org/10.1136/bmj.j5397 https://doi.org/10.1136/bmj.j5397) critiquing the clinical practice of Dr Brown Bear, a fictional cartoon GP.
    - CrazyStat 7y ago
      My favorite BMJ Christmas article is a study of self-reported virgin mothers [1], but I might be biased because I know one of the authors. [1] https://www.bmj.com/content/347/bmj.f7102 https://www.bmj.com/content/347/bmj.f7102
  - Ma8ee 7y ago
    Randomized studies of new treatments often exclude the sickest patients that could have benefitet most from the treatment. In particular, there won't be any really seriously ill patients in the control group because they will most likely die without treatment. That skews the results of these studies. This is what they try to illustrate with the jumping out of plane study. The result won't be reliable if you don't have a control group that actually might have benefit from the treatment. Cancer patients (jumpers), will or will not get a new type of chemotherapy (parachute). But since it is known that the sickest (hightest jumping hight) certainly will die without treatment (parachute), they decide to only test the treatment on patients with only early stage and benign cancer (close to the ground).

2stop 7y ago

Research done more than 20 years ago (and repeated and confirmed) already found that teachers have very little effect on educational outcomes. The biggest correlative factor is socioeconomic status and education level of the parents. Everything else is hit or miss But hey, let’s publish a pointless paper... because academia.

dooglius 7y ago
According to the paper, many states are still using VAM so it bears repeating.
sethammons 7y ago
I think this is an important paper. Even or especially if it confirms other studies. When I was a teacher in inner city San Bernardino, we were held accountable for student achievement ignoring student ability, socioeconomic situation, home life, or their capacity to disrupt the class. In large part, this is because of studies that say a teacher has outsized effect on student test scores. Now, I think a special kind of person can connect better with some kids and help them achieve. I think that is not something you can mass produce. With more studies showing teachers are not as critical as schools want to believe, maybe we can start focusing on other metrics to gauge suçcess.
- ckcheng 7y ago
  > With more studies showing teachers are not as critical as schools want to believe, maybe we can start focusing on other metrics to gauge suçcess. Like cost of hiring? Seriously, if teachers are not as critical, why not hire less or pay less salary... any half decent person in the classroom will do. I think the study is about right that teachers are not as critical as they'd like to believe. I just don't know what to think of the implications...
  - sethammons 7y ago
    I still think a bad teacher is absolutely terrible and that teaching is not easy. I don't look at teacher salaries and think they are too high at all. I think we are bad at measuring what makes a good teacher. I just know it is not based on standardized test scores. When I was a teacher, I had zero power to remove a disruptive or inattentive student from my class. I had 9th graders who could not deal with negative numbers being expected to do well on state Algebra tests that are of arguable quality. Even if they made substantial progress, they would not be to grade level. I can still remember one student who after nearly a year of not being bothered to pay any shred of attention in class, during the final review, paid enough attention during the last steps of solving an equation: Me: Alright, and combining like terms, what is 10x - 18x? Student, suddenly paying attention: But wait, you can't subtract a bigger number from a smaller number! I can't stop the class on nearly the last week of 9th grade algebra to help this student understand negative numbers. And they are not interested in coming in to get help outside of normal class. Parents were not interested in making sure this kid got through school. Not so fun fact: at this school, less than 4% would go onto any post-secondary education. Of them, about 2% would go on to finish a degree. Something like a 90% transitory rate (meaning that most students who started 9th grade at this school would transfer to a different school before graduation, if they made it that far). I could go on, but there was an underlying cultural issue that did not value education. That trend is hard to buck: I had third generation gang members, kids raising their siblings because parents (often a single parent) were working multiple jobs, kids dealing with daily violence, one kid was stabbed to death right off of campus. The majority of these kids and their families have no experience seeing what an education can do for their prospects. Add onto that a tough employment market (inner city San Bernardio!), and those who did have siblings who did get a degree, they often couldn't find a job. All this to say, I don't think lowering salary of teachers would help. Maybe lowering the cost of administrators would help. At this same time, there were more administrative personnel in the district than teachers, something I could never understand.

pjc50 7y ago

It's not totally beyond the realm of plausibility for teachers to have long-term effects on height, given the effect of childhood nutrition on height. Properly feeding children can have such a positive effect on their academic achievement that some teachers end up doing it out of their own pocket. Certainly the best way to raise the worst-performing children is "fixing" whatever disaster family situation they are dealing with at home.

deleted 7y ago
[deleted]
lonelappde 7y ago
Teachers are not feeding kids in statistically significant volumes. Schools are, and there might be correlations.
- ChrisLomont 7y ago
  >Teachers are not feeding kids in statistically significant volumes. Likely untrue, given a few moments search. For example, "On average, teachers spend $37 a month and principals spend $59 a month for food for their students." [1]. There's lots more data on teachers feeding kids in perhaps causal amounts. [1] http://neatoday.org/2013/09/10/report-teachers-spend-37-every-month-to-help-feed-hungry-students/ http://neatoday.org/2013/09/10/report-teachers-spend-37-ever...
  - dooglius 7y ago
    How many students do teachers have on average? $37 per month definitely does not seem significant enough to explain the effect found.
    - ChrisLomont 7y ago
      Avg public school teacher:student ratio is 16:1, avg school meal costs $2.48. This may be a non-negligible effect. Given the strong correlation between some breakfast vs none, and school performance, it would not take much to affect the outcomes by feeding hungry kids little snacks.

jotakami 7y ago

My son started kindergarten this year, and unfortunately we are zoned into one of the worst schools in the district. He’s a bright kid, already knows everything they would teach him, so we weren’t concerned about academics, but let me tell you—his behavior at home, and especially his language use, has taken a really troubling turn. He is obviously picking up the behavior of some of the children he goes to school with. I read a book not too long ago entitled “The Nurture Assumption,” by Judith Harris. It influenced me more than anything I’ve read as a parent, because it is so obviously true. Her thesis is that everyone assumes that adults have a significant impact on a child’s future, yet there is basically zero evidence to support such an assumption. Peer groups are the dominant influence on a child’s personality and development, full stop. This is probably hard for educators to accept because there’s really nothing that can be done about it from their perspective.

rimliu 7y ago
So loners are not influenced by anything?
Ididntdothis 7y ago
I don’t understand why people often want to reduce something to only one factor that explains everything . When I look back at my childhood I was influenced positively and negatively by my parents, siblings, teachers, my peers, neighbors and a lot of other people. They all had an influence at times that deeply impacted my life.
- lallysingh 7y ago
  For planning and resource application purposes, you can only sell effort on one factor at a time to non-specialists (policy makers, parents, the public)
  - Ididntdothis 7y ago
    That's how you get terrible planning. Maybe we should just accept that things are not that simple and there are a lot of things that influence complex systems.

bhouston 7y ago

This is also the tale relevant to most A/B testing.

jefftk 7y ago

Summary: schools are rewarding teachers based on "value-added measures" that try to measure the effect of the teacher on the student's educational progress. The paper shows that these models are mostly picking up unrelated differences between student populations, by showing that the same approach would reward teachers for having taller students.

Accujack 7y ago
Oddly, most school basketball programs DO get rewarded for having taller students.
klipt 7y ago
Our society has decided that taller is better, so naturally the best (tallest) students should get the best (tallest) teachers!
- Spooky23 7y ago
  My son’s teacher is 4’11”. I’m filing a complaint immediately.
wtvanhest 7y ago
Could be a landmark paper. At a minimum, it shows that all evaluation methods should completely negate teacher's impact on height. It seriously calls in to question a ton of research that I would have assumed to have been good. Excellent work by those that wrote the paper.
- datashow 7y ago
  It appears to be an important research study, but in the sense of implication to educational policy, not so much. The reason is, for one, the use of value-added measurement does not have a significant role as many would think. There is not a signle state in the U.S. use VAM as a single measure of teacher performance, nor a determinant factor in teacher firing or promotion. It is always mixed with other quantitative and / or qualitative measures (which usually are no better than VAM). The second reason is, the research community has not find a better replacement yet. We always know we cannot directly use student achievement, now we know we cannot trust a more complicated measure such as VAM which takes into account of student history and other factors.

omarhaneef 7y ago

I think the message to parents is loud and clear: send your kids to the teacher that will make them tallest.

_benj 7y ago
You have made me laugh dear sir or ma'am! :P

nicoburns 7y ago

I'd strongly question our ability to measure educational outcomes. If they're basing that on standardised testing, then I'd argue that pretty much invalidates any conclusions of the research.

dissidents 7y ago
Do you think it's inherently impossible to measure educational outcomes or just difficult with the tools that we today?
- nicoburns 7y ago
  I think it's very difficult to come up with accurate objective measurement methods (nobody's done it well yet). The best assessments tend to be subjective teacher-based assessments (for similar reasons as to why hiring tends to be done by subjective personal assessment - it's hard if not impossible to define the criteria for "good" upfront. Someone may approach things in a completely different way that nobody had thought to measure). And this is before you get into the biases of the people drawing up the assessment criteria etc.

petrogradphilos 7y ago

> 1 Introduction > The increased availability of data linking students to teachers has made it possible to estimate the contribution teachers make to student achievement. There was some data available before (or the sentence would not have used the word "increased"). Why wasn't it possible to estimate with that? > By nearly all accounts, this contribution is large. It goes on to talk about what "large" means: > Estimates of the impact of a one standard deviation (σ) increase in teacher “value-added” on math and reading achievement typically range from 0.10 to 0.30σ, which suggest that a student assigned to a more effective teacher will experience nearly a year's more learning than a student assigned to an less effective teacher (Hanushek & Rivkin 2010;...). (Typo: "an less effective" should be "a less effective".) A "range from 0.10 to 0.30σ" doesn't make sense. A Greek lowercase sigma (σ) is used to represent one standard deviation, but the sigma is used only on the upper end of the range. Should it have been from 0.10σ to 0.30σ? And how are they measuring the impact on achievement of an increase in teacher "value-added", anyway? It says that estimates of the impact "typically range from 0.10 to 0.30σ", but it doesn't say what units those figures are in. The sentence goes on to say that those unit-less estimates "suggest" that "a student assigned to a more effective teacher will experience nearly a year's more learning than a student assigned to an less effective teacher". Over what time period? That is, how long does a student have to study under a "more effective teacher" to get "a year's more learning"? 1 week? 12 years? It doesn't say. And finally, how do those unit-less estimates "suggest" an impact measured in learning time? It doesn't say.

6gvONxR4sf7o 7y ago
You are being absurdly pedantic. For example >A "range from 0.10 to 0.30σ" doesn't make sense. Out loud, you might read it as "oh point one to oh point three sigma." And it's fine. Ever written a phrase like "it's very expensive, costing somewhere around $3-4M" with one "$" and one "M" but two numbers?
Mathnerd314 7y ago
NYC data goes back to 1999, but only for some basic things like test scores. They probably don't have the consolidated links between students and teachers outside of the years they analyzed, due to differing data formats and general laziness. It's a terrible paper in general though, for example it references Bitler et al. 2015 but never provides a full citation.

brohee 7y ago

Can't econonomists learn LaTeX? The typesetting is so horrible its putting you off from reading it...

deleted 7y ago

[deleted]

deleted 7y ago

[deleted]

snarf21 7y ago

"When a measure becomes a target, it ceases to be a good measure." - Goodhart's Law

jeffdavis 7y ago

If somebody pays for something, they want to see the results. If the effects are local, you see with your eyes and you hear things second hand; a kind of soft feedback. If the effects are far away, you probably want to see a hard number, because you don't get the same kind of soft feedback. Is the number (teacher score) useful? Maybe not. But you can't simply take away a number, resulting in no feedback. You have to either come up with a new number that is useful, or localize it enough that people can see the results. Otherwise people won't want to pay. That's why these weird bureaucratic numbers usually come from state or federal requirements. People see the big pile of money going somewhere and they want "accountability" because they can't see the results for themselves. Let's stop shuffling around so much money, laundering it through state and federal agencies. The money, if it comes back, always comes with strings attached. This goes for schools, highway funds, and many other things.

jedberg 7y ago

The problem with using height is that there is already evidence that taller people are more successful due to hidden biases. So perhaps the teacher measures were still valid because all the tall people were more successful because of other biases in the system? I'm mostly just being cheeky here. To be clear, I still think it's nearly impossible to measure how good a teacher is based on student performance, since the teacher is only one part of the equation, the parents (and the student's home life) being a much bigger part of the equation. The teacher can only do so much if the child comes home to parents that don't feed them and tell them they are worthless.

telotortium 7y ago
Perhaps, but the point is that the value-added statistics that have been used (in trials, I don't know if this was ever into a formal metric with consequence) to measure how much students' grades or test scores are improved by a particular teacher's performance show almost as large of a supposed effect of the teacher on height, not height on grades.
- jedberg 7y ago
  The study says that the measure of teacher performance is correlated to student height. The measure of teacher performance is supposed to be correlated to student success. What I'm saying is that if student success is already correlated to student height, then anything that measures student success will also measure student height.
  - JamesBarney 7y ago
    While what you say so true if they're ere just measuring student height/success. But they're not measuring student height/success they're measuring the effect of teachers on student success/height over time. And what you said isn't true of that measure.
    - jedberg 7y ago
      Student success over time is correlated to student height. It would be impossible to measure one without measuring the other.

MarcScott 7y ago

I worked as a teacher in, state funded, education for 15 years. Value added measurements of student achievement will always fail. In affluent areas, where student achievement is already high, schools can afford to hire the best teachers, but they will realistically only be able to boost achievement by a small fraction, given the child's prior attainment. In deprived areas, where student achievement is lower, schools can only afford to hire less qualified teachers, and the boost to achievement will be a small fraction of previous attainment. Either way, the measured effect that the teacher has will be marginal, while common standards and common testing are the only measured outcomes. This does not equate to the teachers not being invested in student outcomes, or trying their best for their students. Value added measures were only introduced as a form of target setting, for teacher performance. I'd encourage everyone to watch Adam Curtis' The Trap[1] to see the effects of target setting. Teachers in early years education set a target, then primary school teachers need to show value added so game the system to show improvement. This trickles up the system, and the net result is unrealistic targets for almost every child as they exit formal education. [1]https://en.wikipedia.org/wiki/The_Trap_(TV_series) https://en.wikipedia.org/wiki/The_Trap_(TV_series)

jim-greer 7y ago
What do you think is the best way to evaluate teachers, if any?
- germinalphrase 7y ago
  I teach at the high school level. I don’t think we have figured it out yet; however, my personal belief is that it would require the crevasse between the high-volume/low-stakes instruction-focused assessment (which cannot be “trusted“ generally) with standardized assessment (which can be “trusted”)to be bridged in such a manner that the daily assessment that takes place all across the country, every single day, can be viewed as legitimate measures of student capability. Without being able to accurately measure student ability day to day, we will not be able to measure teacher effectiveness. I would love to see a toolset like this come into being, but I have no ability to develop such tools.

sadness2 7y ago

It seems to be universal that basing people's career progression on metrics is harmful. People try to do this with engineers as a substitute for understanding what's happening on the ground and having an actual relationship. At best, metrics can raise a flag for you to ask empowering questions.

meekstro 7y ago

I know a brilliant junior school teacher teaching years 1 to 3. He constantly configures his classroom to maximise learning continuity for the entire class. It is a massive challenge balancing out the problems students face physically or outside the classroom. Teaching has become the easy part of teaching. The problem with student learning is that is affected by three things. Emotional Health Physical Health Teaching Quality to build an educated adult you need to optimise all three variables. The problem with rewarding teachers is that their students aren't objectively assessed on all three dimensions. In every district in the world there should be three person educational assessment teams that consist of an emotional health assessor, a physical health assessor and an academic progress assessor that work as a team to objectively assess the students and work with the teacher and support agencies to eliminate learning barriers. Any students that need help should get referred to the correct service or program as quickly as possible. The teacher should focus purely on teaching and integrating support agencies. Teachers deserve to be rewarded for their teaching ability in an objective way. As it happens there is a lot of unnecessary subjectivity and people blame teachers for not being social workers, counsellors and doctors. The education of every child in a class deteriorates as these three dimensions become less homogenous because the maximised learning continuity of the group depends on a limited resource. It is teachers who can creatively balance them while engagingly delivering the curriculum that are creating unrewarded alpha in the educational space. Paying $250,000 a year in salary to a group of 3 specialist objective assessors with the data linked to the health and social services would get a better return on investment than any other expenditure a government could make in lifting the health, education and future earning potential of children. If a child isn't learning there is always a reason. The faster you address that reason the more value you create. Sometimes it's the teacher, sometimes it isn't but for the child's sake we should definitely diagnose the reason.

deleted 7y ago

[deleted]

neallester 7y ago

For children who are still growing physically and mentally, both height and intellectual development are correlated with age, aren't they?. In an analysis where you are treating children who actually have a range of ages as a single age (i.e. "5th graders") finding some correlation between height and intellectual development isn't that surprising. It is just an expression of the underlying correlation between their age and intellectual development.

xyzzyz 7y ago
Height is generally correlated with IQ, though not very strongly (r = 0.2), and IQ is very strongly correlated with education success. It is also known that this correlation is not spurious: half of it is pleiotropy, which is, some of the genes that cause taller height also cause higher intelligence, and half of it is due to assortative mating, meaning taller people tend to have children with other taller people, so if pleiotropy causes taller people to be more intelligent, the tendency to marry taller people additionally amplifies the correlation. Keep in mind though that the correlation, while very highly statistically significant and replicated over many data sets, is not very strong: at r = 0.2, the r^2 is 0.04, so only 4% of variance in intelligence is explained by height. Thus, it is not clear whether the height-IQ correlation plays significant role as a confounder in the study above.
- iguy 7y ago
  But the numbers you describe are for adult height, right? Wouldn't you expect a much stronger effect if your sample was a mix of (say) 4-6th-grade kids. I think that's what GP is saying.
  - Beldin 7y ago
    Kids do not grow in lock-step. I.e., kids who will have exactly the same height as adults will have (probably considerably) different heights at the same age while growing up. Given that, i doubt height measured at any one (age)point during childhood strongly correlates with iq. You'd need sufficient data to be able to reliably predict height as an adult for that, would be my guess.

mcguire 7y ago

Hypothesis: Teachers, individually, don't have statistically measurable effects. Students in a given cohort are mostly identical, and are mostly exposed to identical environments, such that influences on them average to some mean. As a result, minor influences, as long as they act reasonably consistently, are elevated to statistical significance.

datashow 7y ago
On average, it would be true, meaning most teachers do not have significantly different effect on most students. However, there are always particularly good and bad teachers.

drited 7y ago

Ruling out teacher impact on height without testing is perhaps not strictly scientific? We really don't know what is true unless we test it. I know it sounds ridiculous but it's not beyond the realms of possibility that teacher opinion of student capabilities influences social status which in turn influences release of growth hormones which influence height. Aren't there other species which grow when they rise to the top of the pack as growth hormones are released to assist in maintaining that status? Not saying it's likely but if left untested we cannot rule that out. There have been plenty of surprises in science when we went and actually tested what we previously presumed to be implausible.

youzicha 7y ago
Or perhaps good teachers are also better at getting social services to poor students, so they can get food stamps, eat better, and grow talller? https://xhxhxhx.tumblr.com/post/189336550187/teacher-effects-on-student-achievement-and-height https://xhxhxhx.tumblr.com/post/189336550187/teacher-effects...

droithomme 7y ago

There's a significant positive correlation between height and IQ: https://en.wikipedia.org/wiki/Height_and_intelligence https://en.wikipedia.org/wiki/Height_and_intelligence There's also a correlation between IQ and academic achievement. So, if we did decide for some (undoubtedly bad) reason to reward teachers for happening to have more than average tall students it would be comparable to rewarding them for having more than average high achieving students.

paggle 7y ago

Anyone who has done A/B testing knows that you need tens of thousands of data points to pick up any causal relationship unless it’s a huge effect like the retention impact of telling your customers to lick a donkey’s balls. No way you could pick it up for a teacher with 30 kids a year except over that teachers entire career.