3 ms·

My wife is a science-educated high school teacher, and as part of getting her B.Ed she had to do a literature review. She was alarmed by the lack of rigour in e

My wife is a science-educated high school teacher, and as part of getting her B.Ed she had to do a literature review. She was alarmed by the lack of rigour in education research. Almost all the studies she found in education journals were poorly designed, the biggest flaws being: 1) A lack of control groups. In fact, she was told by her instructors that many journals consider it an ethics violation to use a control group, the logic being that in denying your experimental method to the control group you're putting them at a disadvantage. 2) Overgeneralizing results. It's common to use education students (i.e, future teachers) as experimental subjects in education research. Then the results are interpreted as applying to all students, ignoring all the ways that your typical education students are different from your typical high school or elementary school student. While there were some examples of good education research, she said that by far the best education research she could found was actually published in psychology journals. So it's not surprising to me that myths and even pseudoscience will proliferate in education when the standards on research in the field is so poor.

semi-extrinsic 11y ago
> consider it an ethics violation to use a control group, the logic being that in denying your experimental method to the control group you're putting them at a disadvantage. Oh man. Someone should tell them about double-blind clinical trials for cancer treatment. (Those can be, and sometimes are, "stopped for efficiency", when a midway evaluation shows the treatment is so effective that they already have a statistically significant result. They then start giving the treatment to both groups, I believe.)
- a-priori 11y ago
  I know, right? When my wife mentioned this, that was the first thing I said: there are plenty of fields where the research has life-or-death consequences and they use control groups. Education, by comparison, is low stakes. In medical research they have procedures and statistical techniques to detect significant results and allow for early stoppage. There's no reason the same techniques couldn't be applied to education research. But you need control groups, or you don't actually know that you've made an improvement!
  - theseatoms 11y ago
    > But you need control groups, or you don't actually know that you've made an improvement! Exactly. You're disadvantaging all current and future students by NOT having a control group. </circlejerk>
- dexwiz 11y ago
  But in medicine starting treatment doesn't have to correspond with a calendar event. A 'dose' of education is much larger. It would be very difficult to switch curriculum mid semester, because one is getting better results at the midterm. Talking to teachers, one of the biggest complaints with the last decade of education reforms, is that they have to redo their curriculum every year. Not only what they are teaching changes, but how they are evaluated. Everyone knows that education is broken, and "has a plan to fix it". But their plan is just a new set of arbitrary requirements. They rarely let a system evolve naturally, and instead opt for a yearly rewrite. Its a huge bureaucratic burden that's constantly changing. The IT equivalent would be you need a minimum of 6 hours of training each time a library you use releases an update. The update may require you to use a new language. Using the most up to date version is required by law.
- austinjp 11y ago
  Yup. There are plenty of other trial designs that circumvent the (perfectly valid) objection to control groups being denied treatment. For example, both groups could receive the intervention, just not at the same time. While one group waits the other receives the intervention. The waiting group may be considered a "control". After this period of the trial completes, the waiting group is given the active treatment.
dhimes 11y ago
Your wife might be interested in the What Works Clearinghouse (if she doesn't already know of it). They try to sort through the bullshit. http://ies.ed.gov/ncee/wwc/ http://ies.ed.gov/ncee/wwc/
JacobAldridge 11y ago
"the logic being that in denying your experimental method to the control group you're putting them at a disadvantage." Except when the experimental method fails to meet its objectives, and those in the test group are put at a disadvantage!
- thaumasiotes 11y ago
  Why would a researcher develop an idea for making students worse off? ;p
dbpokorny 11y ago
> many journals consider it an ethics violation to use a control group, the logic being that in denying your experimental method to the control group you're putting them at a disadvantage > doing education science is unethical lel "we're going to teach you how to learn, be we refuse to learn how to teach"
mikekchar 11y ago
Research is incredibly hard with education. Consider the following: all of the people you have met with a university degree have learned, by some definition, the material in their degree. Yet, when we interview people for jobs we clearly find that a very large proportion of computer programmers did not learn the material at school to a level which would allow them to use it. I taught English as a foreign language for 5 years. But I also have the experience of taking more than 10 years of French lessons. I can not, nor have I ever been able to speak French. And yet, I passed more than 10 1-year courses on the subject. In the 5 years that I taught English -- to literally thousands of students, some over the period of 3 years, none of them learned how to speak English as a result of my teaching. Although, I was praised for my ability to teach. I speak Japanese fluently. I learned it on my own as an adult. I have never taken a Japanese course and nobody ever taught me Japanese. And yet, I managed to survive 2 years of married life without once speaking English (my wife has since learned to speak English). So the not-so-trivial question is: how do you measure learning? And how do you assign that learning to different teaching techniques? Not only are the papers generally quite poor, the thing they are trying to measure is incredibly elusive. It is a breeding ground for self deception and outright fraud.

semi-extrinsic 11y ago

> consider it an ethics violation to use a control group, the logic being that in denying your experimental method to the control group you're putting them at a disadvantage. Oh man. Someone should tell them about double-blind clinical trials for cancer treatment. (Those can be, and sometimes are, "stopped for efficiency", when a midway evaluation shows the treatment is so effective that they already have a statistically significant result. They then start giving the treatment to both groups, I believe.)

a-priori 11y ago
I know, right? When my wife mentioned this, that was the first thing I said: there are plenty of fields where the research has life-or-death consequences and they use control groups. Education, by comparison, is low stakes. In medical research they have procedures and statistical techniques to detect significant results and allow for early stoppage. There's no reason the same techniques couldn't be applied to education research. But you need control groups, or you don't actually know that you've made an improvement!
- theseatoms 11y ago
  > But you need control groups, or you don't actually know that you've made an improvement! Exactly. You're disadvantaging all current and future students by NOT having a control group. </circlejerk>
dexwiz 11y ago
But in medicine starting treatment doesn't have to correspond with a calendar event. A 'dose' of education is much larger. It would be very difficult to switch curriculum mid semester, because one is getting better results at the midterm. Talking to teachers, one of the biggest complaints with the last decade of education reforms, is that they have to redo their curriculum every year. Not only what they are teaching changes, but how they are evaluated. Everyone knows that education is broken, and "has a plan to fix it". But their plan is just a new set of arbitrary requirements. They rarely let a system evolve naturally, and instead opt for a yearly rewrite. Its a huge bureaucratic burden that's constantly changing. The IT equivalent would be you need a minimum of 6 hours of training each time a library you use releases an update. The update may require you to use a new language. Using the most up to date version is required by law.
austinjp 11y ago
Yup. There are plenty of other trial designs that circumvent the (perfectly valid) objection to control groups being denied treatment. For example, both groups could receive the intervention, just not at the same time. While one group waits the other receives the intervention. The waiting group may be considered a "control". After this period of the trial completes, the waiting group is given the active treatment.

dhimes 11y ago

Your wife might be interested in the What Works Clearinghouse (if she doesn't already know of it). They try to sort through the bullshit. http://ies.ed.gov/ncee/wwc/ http://ies.ed.gov/ncee/wwc/

JacobAldridge 11y ago

"the logic being that in denying your experimental method to the control group you're putting them at a disadvantage." Except when the experimental method fails to meet its objectives, and those in the test group are put at a disadvantage!

thaumasiotes 11y ago
Why would a researcher develop an idea for making students worse off? ;p

dbpokorny 11y ago

> many journals consider it an ethics violation to use a control group, the logic being that in denying your experimental method to the control group you're putting them at a disadvantage > doing education science is unethical lel "we're going to teach you how to learn, be we refuse to learn how to teach"

mikekchar 11y ago

Research is incredibly hard with education. Consider the following: all of the people you have met with a university degree have learned, by some definition, the material in their degree. Yet, when we interview people for jobs we clearly find that a very large proportion of computer programmers did not learn the material at school to a level which would allow them to use it. I taught English as a foreign language for 5 years. But I also have the experience of taking more than 10 years of French lessons. I can not, nor have I ever been able to speak French. And yet, I passed more than 10 1-year courses on the subject. In the 5 years that I taught English -- to literally thousands of students, some over the period of 3 years, none of them learned how to speak English as a result of my teaching. Although, I was praised for my ability to teach. I speak Japanese fluently. I learned it on my own as an adult. I have never taken a Japanese course and nobody ever taught me Japanese. And yet, I managed to survive 2 years of married life without once speaking English (my wife has since learned to speak English). So the not-so-trivial question is: how do you measure learning? And how do you assign that learning to different teaching techniques? Not only are the papers generally quite poor, the thing they are trying to measure is incredibly elusive. It is a breeding ground for self deception and outright fraud.