4 ms·

Bloom's 2 sigma problem

57 points by gballan 8y ago

imranq 8y ago
The key thing that self studiers don’t realize is that you don’t know what you don’t know. And frequently those things are very subtle but build up over time. Luckily with the internet, any motivated student can find great teachers to help them.
- hef19898 8y ago
  That's true, and the resulting basic mistakes can be extremely hard to get rid off. Like bad foot work in boxing, once you do it you have ahrd time correcting it later.
  - imranq 8y ago
    That’s a good example. Piano is also a classic foundational problem
hef19898 8y ago
That rings true. At one point I came to the conclusion that high attrition rates are first and for most an indication for lacking training / teaching quality and not difficulty of the course or trainjng program. As one (former?) Pararescue instructor stated, 30% always drop out, 10% always pass and the rest is up to the instructor (unfortunately I lost the source for that...).
ulucs 8y ago
I think the more exciting takeaway from this is the Mastery Learning method causes a one sigma gain (from 50th to 84th percentile) for free whereas private tutoring lets us gain another sigma (from 84th to 98th) for 10x-15x the tutoring cost. There’s also something else that’s driving my skepticism further. We know that deliberate practice works [1], and the possible effect size is much larger than two sigmas (think more like from a good chess player to grandmaster [2]). However, at this point the effects of both deliberate practice and tutoring start to blend in too much. The mastery learning aspect already seems like a methodology similar to deliberate practice. Does tutoring cause the additional sigma by letting more deliberate practice happen, or by some other vices? If we’re stuck with a 30-pupil classroom as a tutor, is it possible to extract those vices from private tutoring to help our class succeed? [1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3102783/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3102783/ [2] http://slatestarcodex.com/2017/07/31/book-review-raise-a-genius/ http://slatestarcodex.com/2017/07/31/book-review-raise-a-gen...
- pmiller2 8y ago
  Mastery learning is not costless. The cost is time: time for the student and time for the teacher.
acjohnson55 8y ago
I was a teacher. Before doing my teacher prep program, I tried to gain relevant experience tutoring after school. When I actually started teaching, I realized very quickly that it's a completely different ballgame. Teaching isn't just a one way upload of knowledge. It's an interactive process and feedback from the learner is extremely important. Not just for correcting misconceptions, but also for motivation, partnership, and dialog. One-on-one, the loop is very tight. That's one reason I'm skeptical about technology making a huge impact any time soon. Our technology is just really, really far away from being able to stimulate tutoring.
- sitkack 8y ago
  Technology as a replacement, no; as an augmentation, definitely. A computer could be a powerful tool in improving tutors effectiveness. I just saw a paper on using DNN for generating questions from text which could be used for interactive flash cards (spaced repetition, quizes, etc).
  - acjohnson55 8y ago
    Even for augmentation, I'm bearish, short term. We'll eventually get it right, but not before any kids I have are through schooling.
austenallred 8y ago
We have implemented this methodology at Lambda School (YC S17 - https://lambdaschool.com https://lambdaschool.com) with staggering results. We only get paid if our students are hired making more than $50k/yr as a software engineer or data scientist, so we’re more motivated than most to try things like this, even if they’re expensive. First, mastery-based progression. At the end of every week we have a “sprint challenge,” where a student builds a project using everything that was learned that week. If they didn’t grasp the material, instead of pushing them forward, they stay back and repeat that week until they have mastered it. Of course, this is very expensive to do, and therefore will never be done in most classroom environments (you erode economies of scale), but we find that it’s worth it. The results from that alone have been astounding. We don’t have a scientifically-designed study that shows a standard deviation increase (too many other variables are simultaneously changing), but the difference in student competence and ability at graduation was jaw-dropping. The next aspect is probably an order of magnitude more expensive to implement - individual tutoring. We obviously can’t pair every student with a tutor all day (our instructors are experienced engineers and therefore very expensive), but we keep every student in a group of 8 students with 1 TA (a student who is at least 4 months ahead, performed well, and stays on for a short contract). Importantly those students are happy to be paid to review the material, and cost about 1/5 of what an experienced instructor does. In such a small setting a TA can be there to individually work with a student (though for a little less than an hour a day). Again, I don’t have any good way to scientifically prove this, but the level of ability our students have now at the end of our 30-week curriculum (which can become longer due to mastery-based progression) is incredible. If you’re an engineering or hiring manager and want to try it out shoot an email to hiringpartners@lambdaschool.com and ask to interview a student who had never programmed before the 30 weeks to see what they can do. I thoroughly believe Bloom’s 2 Sigma Problem is the most important study done in education... ever.
- robbiep 8y ago
  How can you have great results but not be able to prove it? I see this perception coming up again and again, and it has, in my experience, always been due to 2 factors: the time has not been taken to determine a study design; or the improvement doesn't exist. I'm not saying your method doesn't work, but surely if it is your business you could take the time and money to prove it (i'm in a similar situation with mine, although in the health space no-one trusts you unless you can publish, which raises a whole other set of problems - ie shitty studies)
andres0l1 8y ago
Bloom’s 2 sigma problem was a fascinating study. Regarding the tutoring aspect, at College Connections we want to take this a step further. We use our in-house-designed software MentorMatch to connect students and tutors based on the strength of their personality compatibility. It has been said that “The mind is not a vessel to be filled, but a fire to be kindled.” This echoes anecdotal evidence that students enjoy their work more when they get on with their teacher. Furthermore, we will reduce the cost of private tutoring by targeting schools recent alumni who are happy to tutor students in their schools for a lower rate than professional private tutors. We are currently hoping to conduct a large scale study with few schools to see whether personality-based tutoring matching can actually yield results beyond two sigma! If you want to know more, check this out: https://medium.com/@maxdunhill/matchtech-white-paper-4a6e4e05a93e https://medium.com/@maxdunhill/matchtech-white-paper-4a6e4e0...

57 points by gballan 8y ago

The key thing that self studiers don’t realize is that you don’t know what you don’t know. And frequently those things are very subtle but build up over time. Luckily with the internet, any motivated student can find great teachers to help them.

hef19898 8y ago
That's true, and the resulting basic mistakes can be extremely hard to get rid off. Like bad foot work in boxing, once you do it you have ahrd time correcting it later.
- imranq 8y ago
  That’s a good example. Piano is also a classic foundational problem

hef19898 8y ago

That rings true. At one point I came to the conclusion that high attrition rates are first and for most an indication for lacking training / teaching quality and not difficulty of the course or trainjng program. As one (former?) Pararescue instructor stated, 30% always drop out, 10% always pass and the rest is up to the instructor (unfortunately I lost the source for that...).

ulucs 8y ago

I think the more exciting takeaway from this is the Mastery Learning method causes a one sigma gain (from 50th to 84th percentile) for free whereas private tutoring lets us gain another sigma (from 84th to 98th) for 10x-15x the tutoring cost. There’s also something else that’s driving my skepticism further. We know that deliberate practice works [1], and the possible effect size is much larger than two sigmas (think more like from a good chess player to grandmaster [2]). However, at this point the effects of both deliberate practice and tutoring start to blend in too much. The mastery learning aspect already seems like a methodology similar to deliberate practice. Does tutoring cause the additional sigma by letting more deliberate practice happen, or by some other vices? If we’re stuck with a 30-pupil classroom as a tutor, is it possible to extract those vices from private tutoring to help our class succeed? [1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3102783/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3102783/ [2] http://slatestarcodex.com/2017/07/31/book-review-raise-a-genius/ http://slatestarcodex.com/2017/07/31/book-review-raise-a-gen...

pmiller2 8y ago
Mastery learning is not costless. The cost is time: time for the student and time for the teacher.

acjohnson55 8y ago

I was a teacher. Before doing my teacher prep program, I tried to gain relevant experience tutoring after school. When I actually started teaching, I realized very quickly that it's a completely different ballgame. Teaching isn't just a one way upload of knowledge. It's an interactive process and feedback from the learner is extremely important. Not just for correcting misconceptions, but also for motivation, partnership, and dialog. One-on-one, the loop is very tight. That's one reason I'm skeptical about technology making a huge impact any time soon. Our technology is just really, really far away from being able to stimulate tutoring.

sitkack 8y ago
Technology as a replacement, no; as an augmentation, definitely. A computer could be a powerful tool in improving tutors effectiveness. I just saw a paper on using DNN for generating questions from text which could be used for interactive flash cards (spaced repetition, quizes, etc).
- acjohnson55 8y ago
  Even for augmentation, I'm bearish, short term. We'll eventually get it right, but not before any kids I have are through schooling.

austenallred 8y ago

We have implemented this methodology at Lambda School (YC S17 - https://lambdaschool.com https://lambdaschool.com) with staggering results. We only get paid if our students are hired making more than $50k/yr as a software engineer or data scientist, so we’re more motivated than most to try things like this, even if they’re expensive. First, mastery-based progression. At the end of every week we have a “sprint challenge,” where a student builds a project using everything that was learned that week. If they didn’t grasp the material, instead of pushing them forward, they stay back and repeat that week until they have mastered it. Of course, this is very expensive to do, and therefore will never be done in most classroom environments (you erode economies of scale), but we find that it’s worth it. The results from that alone have been astounding. We don’t have a scientifically-designed study that shows a standard deviation increase (too many other variables are simultaneously changing), but the difference in student competence and ability at graduation was jaw-dropping. The next aspect is probably an order of magnitude more expensive to implement - individual tutoring. We obviously can’t pair every student with a tutor all day (our instructors are experienced engineers and therefore very expensive), but we keep every student in a group of 8 students with 1 TA (a student who is at least 4 months ahead, performed well, and stays on for a short contract). Importantly those students are happy to be paid to review the material, and cost about 1/5 of what an experienced instructor does. In such a small setting a TA can be there to individually work with a student (though for a little less than an hour a day). Again, I don’t have any good way to scientifically prove this, but the level of ability our students have now at the end of our 30-week curriculum (which can become longer due to mastery-based progression) is incredible. If you’re an engineering or hiring manager and want to try it out shoot an email to hiringpartners@lambdaschool.com and ask to interview a student who had never programmed before the 30 weeks to see what they can do. I thoroughly believe Bloom’s 2 Sigma Problem is the most important study done in education... ever.

robbiep 8y ago
How can you have great results but not be able to prove it? I see this perception coming up again and again, and it has, in my experience, always been due to 2 factors: the time has not been taken to determine a study design; or the improvement doesn't exist. I'm not saying your method doesn't work, but surely if it is your business you could take the time and money to prove it (i'm in a similar situation with mine, although in the health space no-one trusts you unless you can publish, which raises a whole other set of problems - ie shitty studies)

andres0l1 8y ago

Bloom’s 2 sigma problem was a fascinating study. Regarding the tutoring aspect, at College Connections we want to take this a step further. We use our in-house-designed software MentorMatch to connect students and tutors based on the strength of their personality compatibility. It has been said that “The mind is not a vessel to be filled, but a fire to be kindled.” This echoes anecdotal evidence that students enjoy their work more when they get on with their teacher. Furthermore, we will reduce the cost of private tutoring by targeting schools recent alumni who are happy to tutor students in their schools for a lower rate than professional private tutors. We are currently hoping to conduct a large scale study with few schools to see whether personality-based tutoring matching can actually yield results beyond two sigma! If you want to know more, check this out: https://medium.com/@maxdunhill/matchtech-white-paper-4a6e4e05a93e https://medium.com/@maxdunhill/matchtech-white-paper-4a6e4e0...