5 ms·

Why do we have such terrible evidence on homework effectiveness? (2022)

24 points by barry-cotter 3y ago

taeric 3y ago
I thought there were studies indicated in Peak on structuring school differently. Thought there were indications from flight schools, but I can't remember details. :( I'm assuming none of those have held up?
alephnerd 3y ago
[flagged]
- ShamelessC 3y ago
  You aren't alone.
  - alephnerd 3y ago
    And I got flagged for calling out the reductiveness of LessWrong style arguments. Really touched a nerve apparently.
    - ShamelessC 3y ago
      Yeah they’re fond of flagging in my experience. It’s kind of a cult for the emotionally unintelligent and/or wealthy programmer. Lots of lonely people in need of a belief system and finding one in statistics, longtermism, techno-optimism, techno-doomerism, and Spock/Sherlocke style “Bayesian” reasoning. Much of the research done is very overconfident (and reductive, as you mention). They make very bold assumptions, casually throw out contradictory results as outliers, make massive leaps to hypothesis which sound more like science fiction than science. They personally really creep me out. The personality types it attracts seem really impressionable and lacking in a healthy disrespect for authority. Also seem to be a lot of Wolfram types who simply ignore criticism rather than admitting a mistake. Fond of the sound of their own voice.
      - alephnerd 3y ago
        Agreed. I mean there value in using hard rationalism in the social sciences, but science is inherently built on a model. Models are sometimes wrong (they're abstractions and approximations - there's a reason we use confidence intervals), but that doesn't mean it's all for naught - we learn something new and continue to iterate. (Also, it feels like people in the LessWrong space never really heard about econometrics or the UChicago/Fresh Water School of Social Sciences - ie using quantitative methods in the social sciences space, which is something they've along with CMU pushed since the 30s) The entire movement feels like people who read Asimov's Foundation, but ignore the fact that even the Foundation failed to predict the Mule. It also feels like the ETO in the 3 Body Problem (the Tencent version, never watched the netflix one and imo never will)
- creesch 3y ago
  The article is outright wrong, there has been extensive research on the effectiveness of homework. It is quite well understood what makes homework effective, that understanding is also rarely applied in practice as it requires a lot of effort. This also makes the easiest way to make homework effective is to give homework to students. Simple as that.
- deleted 3y ago
  [deleted]
MattGaiser 3y ago
Is there any good evidence on how to structure learning really? It all seems quite haphazard, to the point that even in the same school the same subject can be taught very differently depending on the teacher. Anecdotally, teachers seem thrown to the wolves regarding materials or what works best.
- creesch 3y ago
  There is actually plenty of research available about didactic methods in general. The problem I am seeing is that a lot of these are just outright ignored by a large part of the education system. For a variety of reasons ranging from budget to simply not wanting to put in the extra effort.
creesch 3y ago
> Nobody Knows How Well Homework Works Huh, interesting I actually did my graduation thesis on this when I finished my education degree. There is actually some well established literature about how effective homework can be. I wrote about it years ago on reddit: https://old.reddit.com/r/history/comments/jkt6l3/no_we_really_are_not_going_to_do_your_homework/gal5ovm/ https://old.reddit.com/r/history/comments/jkt6l3/no_we_reall... Cliffs notes and sources: - For homework to be effective teachers need to give it with a clear communicated goal and make sure to also discuss the work done is discussed afterwards to reinforce meaningfulness. - There is a clear limit on the amount of time students can spend on homework before it starts to have diminishing and even negative returns. - Ideally there is a school wide policy where teachers are aware of the amount of work other teachers give and make sure the combined work doesn't exceed the threshold. - Home situation matters a lot. tl;dr a clean quiet place to work is best. - Parents role is important but not in the way they often think. The role of parents lies in "facilitating" the homework by, for example, offering the child a good place to study, helping to monitor time. However, parents have to be careful not to solve problems for their child. All of these points need to be met in order for homework to have a reasonable chance of being effective. * Cooper, H. a. V., J. C. (2001). Using Research to Answer Practical Questions About Homework. Education psychologist 36(3), 11. * Donk van der, C., Lanen van, B. (2009). Praktijkonderzoek in de school. Bussum: Coutinho. * Geerlings, T., Veen van der, T. (2008). Lesgeven en zelfstandig leren. (12e druk). Assen: Van Gorcum. * Marzano, R. J. a. P., D.J. (2007). The Case For and Against Homework. Educational leadership 64(6), 6. * Marzano, R. J., Pickering D. J., Pollock J. E.(2001). Classroom instruction that works: research-based strategies for increasing student achievement. Alexandria, Virgina: ASCD. * Neuwahl, N.M.E., Bogaart, P.H.M. de. (1989). Huiswerk. Leiden: Martinus Nijhoff. * Rosmalen, M. van, Dam, J van, Struijk, A., Vermeer, E. (2009). Wat maakt huiswerk nuttig?. Master thesis. Utrecht: Instituut voor Lerarenopleiding, Onderwijsontwikkeling en Studievaardigheden (IVLOS). * Trautwein, U. (2007). The homework-achievement relation reconsidered: Differentiating homework time, homework frequency, and homework effort. Learning and instruction 17(3), 372-388. * Trautwein, U. a. K., Olaf (2003). The relationship Between Homework an Achievement - Still Much of a Mystery. Educational psychology review 15(2), 115-145.
- aardvarkr 3y ago
  Thank you for the comprehensive answer! I was highly skeptical that there was no good evidence on the effectiveness of homework.
deleted 3y ago
[deleted]
zero-sharp 3y ago
? What is the purpose of the homework? What is homework? Is it literally just work? Does it take into account an individual student's needs? If it doesn't, then it's pretty obvious why that would be ineffective. Not all people are the same and, as a result, not all work has the same effect. I'm sure somebody has thought about this before. But honestly, this line of research kinda reminds me of the conversation over weight loss and calories. Have you tried eating less? Have you tried studying more? Are we waiting for formal research to tell us that putting in work might lead to better results? Yes, there are individual differences. But unless you know you're an exception, you're probably not. And it not being perfect doesn't mean it's not a good policy.
- Fire-Dragon-DoL 3y ago
  It's practice usually. Repeat something over and over to figure out edgecases
  - zero-sharp 3y ago
    There's many different things we might expect students to gain from homework. One might be: building memories and familiarity. Another might be: practicing problem solving. There's plenty of work that teachers assign that do some or none of those things. When I was doing my math homework in high school, I really wasn't concerned about edge cases. That's work at a higher level than most K-12 students aim for.
    - Fire-Dragon-DoL 3y ago
      I mean, when I was doing homeworks in high school all I was concerned about was finishing it in the shortest time possible, everything else was irrelevant, lol
deleted 3y ago
[deleted]
toast0 3y ago
IMHO, education research and diet research suffer the same basic problem that different people respond to different approaches, but studies are looking for a single approach that works for everyone. They're also hard studies to run because compliance is hard to measure at the timescales involved. I'd love to see results more like these N methods will cover most people, and here's what you can do to identify which method is best for a person, and how to switch. But also, in education, it's not easy to provide multiple approaches for learning to different students in the same classroom, and it's not easy to provide very small classsizes so that many approaches are available.
- moffkalast 3y ago
  The solution isn't to provide multiple approaches in the same classroom, it's to remove the classroom altogether. I keep coming back to CGP Grey's take[0] on the issue, and since he's an ex-teacher it makes him somewhat qualified on the topic. Conceptually the problem is solved, it's one-on-one tutoring that was widely practiced in the ancient world for the aristocracy. The question is how to effectively implement that for the masses. Khan academy has attempted to use GPT 4 as a teacher bot in much this manner (to some success?), but the current gen of language models isn't smart enough nor designed to tackle this problem in a holistic manner. In a decade or two though? Easily. Obviously there are numerous problems when it comes to deploying this sort of thing to the public school system. I mean, can you really trust a private company do handle it? Eventually though, something will have to be implemented when the ever dwindling number of underpaid teachers forces our hand. [0] https://www.youtube.com/watch?v=7vsCAM17O-M https://www.youtube.com/watch?v=7vsCAM17O-M
  - anotherhue 3y ago
    Firstly, let us not conflate a full time professional YouTube personality with dedicated researchers, no matter how sexy the algorithm makes him. Secondly, we can demonstrate that this method is some AI hype nonsense because there exist hundreds of books on basic topics like mathematics all written for different learning styles. The LLM is not bringing anything to the table to make up for it's major shortcomings. Most importantly is that the human learning interface for children is far too vast for a simple model to attempt to interact with. Smells, hunger levels, nap phase, whatever. An LLM won't interface with it anymore than a pigeon can interface with a 24fps film. https://www.psychologytoday.com/gb/blog/what-shapes-film/201406/why-you-can-t-take-pigeon-the-movies https://www.psychologytoday.com/gb/blog/what-shapes-film/201...

24 points by barry-cotter 3y ago

I thought there were studies indicated in Peak on structuring school differently. Thought there were indications from flight schools, but I can't remember details. :( I'm assuming none of those have held up?

alephnerd 3y ago

[flagged]

ShamelessC 3y ago
You aren't alone.
- alephnerd 3y ago
  And I got flagged for calling out the reductiveness of LessWrong style arguments. Really touched a nerve apparently.
  - ShamelessC 3y ago
    Yeah they’re fond of flagging in my experience. It’s kind of a cult for the emotionally unintelligent and/or wealthy programmer. Lots of lonely people in need of a belief system and finding one in statistics, longtermism, techno-optimism, techno-doomerism, and Spock/Sherlocke style “Bayesian” reasoning. Much of the research done is very overconfident (and reductive, as you mention). They make very bold assumptions, casually throw out contradictory results as outliers, make massive leaps to hypothesis which sound more like science fiction than science. They personally really creep me out. The personality types it attracts seem really impressionable and lacking in a healthy disrespect for authority. Also seem to be a lot of Wolfram types who simply ignore criticism rather than admitting a mistake. Fond of the sound of their own voice.
    - alephnerd 3y ago
      Agreed. I mean there value in using hard rationalism in the social sciences, but science is inherently built on a model. Models are sometimes wrong (they're abstractions and approximations - there's a reason we use confidence intervals), but that doesn't mean it's all for naught - we learn something new and continue to iterate. (Also, it feels like people in the LessWrong space never really heard about econometrics or the UChicago/Fresh Water School of Social Sciences - ie using quantitative methods in the social sciences space, which is something they've along with CMU pushed since the 30s) The entire movement feels like people who read Asimov's Foundation, but ignore the fact that even the Foundation failed to predict the Mule. It also feels like the ETO in the 3 Body Problem (the Tencent version, never watched the netflix one and imo never will)
creesch 3y ago
The article is outright wrong, there has been extensive research on the effectiveness of homework. It is quite well understood what makes homework effective, that understanding is also rarely applied in practice as it requires a lot of effort. This also makes the easiest way to make homework effective is to give homework to students. Simple as that.
deleted 3y ago
[deleted]

MattGaiser 3y ago

Is there any good evidence on how to structure learning really? It all seems quite haphazard, to the point that even in the same school the same subject can be taught very differently depending on the teacher. Anecdotally, teachers seem thrown to the wolves regarding materials or what works best.

creesch 3y ago
There is actually plenty of research available about didactic methods in general. The problem I am seeing is that a lot of these are just outright ignored by a large part of the education system. For a variety of reasons ranging from budget to simply not wanting to put in the extra effort.

creesch 3y ago

> Nobody Knows How Well Homework Works Huh, interesting I actually did my graduation thesis on this when I finished my education degree. There is actually some well established literature about how effective homework can be. I wrote about it years ago on reddit: https://old.reddit.com/r/history/comments/jkt6l3/no_we_really_are_not_going_to_do_your_homework/gal5ovm/ https://old.reddit.com/r/history/comments/jkt6l3/no_we_reall... Cliffs notes and sources: - For homework to be effective teachers need to give it with a clear communicated goal and make sure to also discuss the work done is discussed afterwards to reinforce meaningfulness. - There is a clear limit on the amount of time students can spend on homework before it starts to have diminishing and even negative returns. - Ideally there is a school wide policy where teachers are aware of the amount of work other teachers give and make sure the combined work doesn't exceed the threshold. - Home situation matters a lot. tl;dr a clean quiet place to work is best. - Parents role is important but not in the way they often think. The role of parents lies in "facilitating" the homework by, for example, offering the child a good place to study, helping to monitor time. However, parents have to be careful not to solve problems for their child. All of these points need to be met in order for homework to have a reasonable chance of being effective. * Cooper, H. a. V., J. C. (2001). Using Research to Answer Practical Questions About Homework. Education psychologist 36(3), 11. * Donk van der, C., Lanen van, B. (2009). Praktijkonderzoek in de school. Bussum: Coutinho. * Geerlings, T., Veen van der, T. (2008). Lesgeven en zelfstandig leren. (12e druk). Assen: Van Gorcum. * Marzano, R. J. a. P., D.J. (2007). The Case For and Against Homework. Educational leadership 64(6), 6. * Marzano, R. J., Pickering D. J., Pollock J. E.(2001). Classroom instruction that works: research-based strategies for increasing student achievement. Alexandria, Virgina: ASCD. * Neuwahl, N.M.E., Bogaart, P.H.M. de. (1989). Huiswerk. Leiden: Martinus Nijhoff. * Rosmalen, M. van, Dam, J van, Struijk, A., Vermeer, E. (2009). Wat maakt huiswerk nuttig?. Master thesis. Utrecht: Instituut voor Lerarenopleiding, Onderwijsontwikkeling en Studievaardigheden (IVLOS). * Trautwein, U. (2007). The homework-achievement relation reconsidered: Differentiating homework time, homework frequency, and homework effort. Learning and instruction 17(3), 372-388. * Trautwein, U. a. K., Olaf (2003). The relationship Between Homework an Achievement - Still Much of a Mystery. Educational psychology review 15(2), 115-145.

aardvarkr 3y ago
Thank you for the comprehensive answer! I was highly skeptical that there was no good evidence on the effectiveness of homework.

deleted 3y ago

[deleted]

zero-sharp 3y ago

? What is the purpose of the homework? What is homework? Is it literally just work? Does it take into account an individual student's needs? If it doesn't, then it's pretty obvious why that would be ineffective. Not all people are the same and, as a result, not all work has the same effect. I'm sure somebody has thought about this before. But honestly, this line of research kinda reminds me of the conversation over weight loss and calories. Have you tried eating less? Have you tried studying more? Are we waiting for formal research to tell us that putting in work might lead to better results? Yes, there are individual differences. But unless you know you're an exception, you're probably not. And it not being perfect doesn't mean it's not a good policy.

Fire-Dragon-DoL 3y ago
It's practice usually. Repeat something over and over to figure out edgecases
- zero-sharp 3y ago
  There's many different things we might expect students to gain from homework. One might be: building memories and familiarity. Another might be: practicing problem solving. There's plenty of work that teachers assign that do some or none of those things. When I was doing my math homework in high school, I really wasn't concerned about edge cases. That's work at a higher level than most K-12 students aim for.
  - Fire-Dragon-DoL 3y ago
    I mean, when I was doing homeworks in high school all I was concerned about was finishing it in the shortest time possible, everything else was irrelevant, lol

deleted 3y ago

[deleted]

toast0 3y ago

IMHO, education research and diet research suffer the same basic problem that different people respond to different approaches, but studies are looking for a single approach that works for everyone. They're also hard studies to run because compliance is hard to measure at the timescales involved. I'd love to see results more like these N methods will cover most people, and here's what you can do to identify which method is best for a person, and how to switch. But also, in education, it's not easy to provide multiple approaches for learning to different students in the same classroom, and it's not easy to provide very small classsizes so that many approaches are available.

moffkalast 3y ago
The solution isn't to provide multiple approaches in the same classroom, it's to remove the classroom altogether. I keep coming back to CGP Grey's take[0] on the issue, and since he's an ex-teacher it makes him somewhat qualified on the topic. Conceptually the problem is solved, it's one-on-one tutoring that was widely practiced in the ancient world for the aristocracy. The question is how to effectively implement that for the masses. Khan academy has attempted to use GPT 4 as a teacher bot in much this manner (to some success?), but the current gen of language models isn't smart enough nor designed to tackle this problem in a holistic manner. In a decade or two though? Easily. Obviously there are numerous problems when it comes to deploying this sort of thing to the public school system. I mean, can you really trust a private company do handle it? Eventually though, something will have to be implemented when the ever dwindling number of underpaid teachers forces our hand. [0] https://www.youtube.com/watch?v=7vsCAM17O-M https://www.youtube.com/watch?v=7vsCAM17O-M
- anotherhue 3y ago
  Firstly, let us not conflate a full time professional YouTube personality with dedicated researchers, no matter how sexy the algorithm makes him. Secondly, we can demonstrate that this method is some AI hype nonsense because there exist hundreds of books on basic topics like mathematics all written for different learning styles. The LLM is not bringing anything to the table to make up for it's major shortcomings. Most importantly is that the human learning interface for children is far too vast for a simple model to attempt to interact with. Smells, hunger levels, nap phase, whatever. An LLM won't interface with it anymore than a pigeon can interface with a 24fps film. https://www.psychologytoday.com/gb/blog/what-shapes-film/201406/why-you-can-t-take-pigeon-the-movies https://www.psychologytoday.com/gb/blog/what-shapes-film/201...