12 ms·
Bloom's 2 sigma problem
- precompute 4y agoSimilar to the "2 Sigma humor gap". Also related: sharply increasing odds of social maladjustment / mistreatment after a certain IQ score.
- hgsgm 4y agoIt's completely different. The only similarity is that a standard deviation is involved.
- precompute 4y agoThe 2 Sigma problem requires a tutor within 2 Sigma.
- Soverinowu689 4y ago[dead]
- swyx 4y agotheres a lot of hope in the educational AI space that we can do personalized teaching and essentially raise everybody to the 90+ percentile implied by the 2 sigma problem (see https://lspace.swyx.io/p/chatgpt-gpt4-hype-and-building-llm https://lspace.swyx.io/p/chatgpt-gpt4-hype-and-building-llm 20 min mark - disclaimer its my podcast with openai's new representative) i think this is a "last mile" type situation where we've seen great progress with ChatGPT on the first 90%, but the last mile will take the remaining 90% to do. not impossible, just won't come anywhere as soon or as easily as people want. still worth pursuing ofc, but not as a quick hustle but more if you are just intensely insightful and passionate about education and figure out how to apply AI to personalize it properly
- bee_rider 4y agoSince ChatGPT often writes untrue statements, it seems like quite a poor teacher. I guess it could be used as an assist for an instructor (send me a question, chat GPT will answer it, I’ll check the output) but that seems not so revolutionary.
- isitmadeofglass 4y ago[dead]
- ohhhhhh 4y ago"too costly for most societies to bear on a large scale" this is changing as we speak with language models.
- c0brac0bra 4y agoMakes me think of the book in Diamond Age.
- hgsgm 4y agoWhat's that book called?
- dredmorbius 4y ago"A Young Lady's Illustrated Primer", which also happens to be the subtitle of The Diamond Age (by Neal Stephenson). It's an interactive e-book, of sorts.
- pantalaimon 4y agoA Young Lady's Illustrated Primer
- nullc 4y ago"A Young Lady's Illustrated Primer"
- yboris 4y agoEducation Automation: Freeing the scholar to return to his studies by Buckminster Fuller (1964) https://www.amazon.com/Education-Automation-Freeing-scholar-studies/dp/0809301377 https://www.amazon.com/Education-Automation-Freeing-scholar-...
- dredmorbius 4y agoThe history of proposed technological replacements for human teachers is filled with highly enthusiastic and widely touted failures. It's not simply a matter of presenting information --- a book, or streamed lecture, or audio or video recording can do this --- or even interactive engagement ("educational software"), but a highly-attentive in-the-moment back-and-forth and with the student, sensing where they do and don't follow or understand a lesson, and most critically allowing them to discover for themselves a principle, concept, or fact. It's not that these cannot happen alone, via self-directed (autodidactic) study, or with the assistance of technologically-mediated aides. But time and again the effectiveness of such methods falls far short of direct human instruction. LLM AI chatbots can respond to written or verbal cues, to an extent, but as yet lack an underlying pedagogical framework and understanding of the context of a communication. From what I've seen so far of such models, "lacking an underlying framework", the bones and skeleton of a conversation, if you will, which can be subsequently fleshed out, seems to be a general failing of the models, so far. I'm not saying that this cannot emerge, and parts of this may in fact be a relatively simple additional step or advance. Given other advances such as sentiment and mood assessment and analysis they may even be largely in-place. What's left however is the motivation of such efforts, and alignment with the educational mission, again a place where previous technological education programmes have foundered, as most have been coopted by, if not outright conceived as, propaganda and cultural programming tools. (A part of traditional education as well, but far less effectively and uniformly delivered when mediated at 1:15 -- 1:30 or so ratios amongst millions of instructors[1]). It will all but certainly be difficult for corporate-based LLM models to resist such temptations, see the earlier case of Backrub and perverse incentives.[2] ________________________________ Notes: 1. Figure based on an off-the-cuff estimate, though substantiated by US Dept. Ed. statistics, see: <https://nces.ed.gov/fastfacts/display.asp?id=372 https://nces.ed.gov/fastfacts/display.asp?id=372>. 2. "The Anatomy of a Large-Scale Hypertextual Web Search Engine" (1998), Appendix A: <http://infolab.stanford.edu/~backrub/google.html http://infolab.stanford.edu/~backrub/google.html>
- hinkley 4y agoWe need to study what amounts of 1:1 tutoring have positive outcomes. If society can’t afford 1:1 tutoring for everyone, all the time, can we afford 20% of the time?
- 082349872349872 4y agoMy understanding is that the knee of the curve is ~10-12 students. Much cheaper than 1:1 or 1:2, but still more expensive than the 30+ classrooms of my youth.
- zozbot234 4y agoYou can teach large classes efficiently, but it requires a very direct, rote, scripted approach that's exactly what most teachers don't want to use - because fancy education schools have told them that it "demeans" the profession, and they've drunk the Kool-Aid. A key component is to come up with memorable text that the students must be able to repeat word for word when prompted - both as a group and individually. (Problem solving similarly starts with trivial cases that the students must answer on the spot; difficulty is very gradually increased, and there's a lot of retracing and review of earlier steps.) This is as close as you can get to "mastery learning" in a large classroom; keep all students on lockstep and highly engaged, trying to ensure that no one falls through the cracks.
- andrewflnr 4y ago"As close as you can get" may be true, but it still doesn't sound very close. In fact the memorization part sounds profoundly miserable. "A lot of retracing" of stuff that, frankly, I probably understood the first couple times also sounds miserable, and inefficient. So, uh, the kool aid is pretty tasty I guess.
- zozbot234 4y agoIt's not "miserable" if the lesson scripts are designed for it. In fact, the challenge of gradually memorizing "the lesson" until you know it word for word can be playful and engaging.
- trane_project 4y agoI am already working on fixing this: https://github.com/trane-project/trane/ https://github.com/trane-project/trane/ No AI needed. Just an old fashioned depth first search through a graph of skills and dependencies. I made it to help me practice music, but I have been branching out and using it to study math for a few weeks. I find myself saying "just one question more" and then spending another half hour in it. Still needs more material to be useful to other people but it's a solid experience. I learned and memorized how to play most of the notes in the saxophone with good intonation in about a week, as a complete beginner.
- cjblack 4y agoI really like simple, reliable, tools like this - just watched the demo video. I hadn't heard of mastery learning before, but now I'm thinking about trying to map some Excel lessons towards this and see if I can implement it at work. I've found that our youngest employees are coming out of school with little-to-no knowledge of how to work spreadsheet software - even those who are otherwise pretty technically proficient. While programmers tend to eschew tools like Excel, in consulting (or at least the consulting we do) it's critical to being able to understand how a process works before designing an automated solution. Excited to potentially have a way to share that understanding.
- trane_project 4y agoThanks for the feedback. An excel course should be doable. Currently the most general way to write material is to use this: https://trane-project.github.io/generated_courses/knowledge_base.html https://trane-project.github.io/generated_courses/knowledge_... Basically a folder of lessons where the flashcards are pairs of markdown files. There's also another tool to write simple flashcards and lessons in a json file, run a command, and build all those directories and markdown files for you, but I have not written the documentation yet. Here's an example for a course based on a reworking of Euclid's Elements: https://github.com/trane-project/trane-math/blob/master/courses/geometry/euclid_1/course_config.json https://github.com/trane-project/trane-math/blob/master/cour... But plain markdown files for now should get you going.
- 4y ago
- lofatdairy 4y agoAlso discussed in the aristocratic tutoring article posted here: https://erikhoel.substack.com/p/why-we-stopped-making-einsteins https://erikhoel.substack.com/p/why-we-stopped-making-einste... and discussed here: https://news.ycombinator.com/item?id=30698624 https://news.ycombinator.com/item?id=30698624
- dredmorbius 4y agoExcellent article and well-worth reading in context of TF Wikipedia A.
- levzettelin 4y agoWhat's problematic about this? Why is this named a "problem"?
- choxi 4y agoI believe it’s that Bloom’s study implies that 1-on-1 pacing and teaching produces the best learning results, but that Bloom thought it was infeasible to scale to everyone: it’s “too costly for most societies to bear on a large scale”
- dredmorbius 4y agoBeyond that are the equity effects arising where availability of such tutoring primarily to wealthy families means that existing differentials in opportunity and advantage are further multiplied. There are a few potential options that appear in or occur as one reads the literature (the linked Wikipedia article lists several references, beyond Bloom's own work). That includes inflection points in the effect, that is, some advantage occurs with lower degrees of specificity, particularly with instruction at a 1:10 -- 1:15 instructor/student ratio or so. Another option would be to randomly apportion tutors to the general population, or to assign tutors either to particularly talented students (regardless of family wealth/income), or to specific underprivileged populations (to reduce systemic and historic educational gulfs), or simply at random (which avoids numerous social, political, personal, and/or academic faddish biases).
- trane_project 4y agoThat personal tutoring leads to better outcomes is not the problem. That will obviously be the case. The problem is how to bridge the gap in a way more people realize similar benefits.
- em-bee 4y agomontessori seems to be the solution here. it is essentially individual tutoring at scale because the teacher/tutor doesn't need to spend 100% of their time with one child. a lot of time the children are practicing on their own, and also they are learning from older students (the tutor doesn't always need to be the most qualified person around, they just need to be able to teach the material that the student is learning). they also don't move on to the next segment until they master the previous one.
- ultra_nick 4y agoOur startup started working on solving this problem with extremely customized continuous AI Tutoring last year. In his research paper, Bloom says that private tutoring is the most effective learning method, but similar results can be obtained through a combination of scientific learning methods. However, in his time, these methods were too computationally expensive for teachers to implement. Now, we have the technology (LLMs) for everyone to have an accelerated contextual learning experience. We started by tracking mastery learning with graphs, but the effort required was too much for users. Now, we're augmenting the previous system with ML to automatically track and explain new concepts. In the far future, I'm hoping we'll be able to accelerate learning rates by 150%, so that motivated highschool students can graduate with a bachelor's degree. Our first AI Tutor should be out by April. Follow us on LinkedIn to get an alert when we launch! https://www.linkedin.com/company/conceptionary/ https://www.linkedin.com/company/conceptionary/ To learn more, checkout our website: https://conceptionary.app/ https://conceptionary.app/
- bowsamic 4y agoDo we need to fix this? I am not sure that everyone needs a "mastery" of education
- HDMI_Cable 4y agoI wonder how aging populations and large-scale language models are going to change the difficulty of this problem. It seems to me that we're just about to unlock 1-to-1 learning, without actually needing 1 teacher per child. It isn't hard to imagine a language model (or something similar, I'm not in AI myself so I might not have the best idea of the specifics) giving personalized instruction to a student based on a knowledge bank and generative prompts + reinforcement from what the child understands. Add to that aging populations leading to less children and higher worker-child ratios, + automation in other industries, and its plausible to imagine that there could be a large movement towards training new teachers, thereby reducing class sizes. This could increase feedback between the two, and probably reduce behavioural problems as teachers could address them more easily. I guess one problem of this approach is that this 'atomizes' the student, where they only engage with their device with the language model, removing some of the collaborative aspects of modern instructional methods. One easy fix to this is to increase the amount of recess or structured play, thereby increasing socialization. This seems to work in Finland [1]. If advances in AI can make learning more efficient, we can increase the amount of time for unstructured play. --- [1]: https://www.theatlantic.com/education/archive/2014/06/how-finland-keeps-kids-focused/373544/ https://www.theatlantic.com/education/archive/2014/06/how-fi...
- civilized 4y agoIt's a nice dream. I would just caution that we will unlock people claiming that this works and foisting a half-baked version of it on large populations to save money, look modern, and make profits for tech companies... long before we unlock the actually good version.
- ducttapecrown 4y agoAny actually good version of a thing is built on the corpses of the previous bad versions though.
- simonster 4y agoThe bad versions should be developed, but they should not be marketed as a replacement for traditional education.
- deleted 4y ago[deleted]
- l_theanine 4y agoEducation is going to be radically changed when AI can manage to helpfully teach students with a Microsoft Tay situation happening. Very quickly after that, I think a sort of global shift will begin, as schools all around the world begin to see the power of legitimate curriculums, instead of lobotomized "think how I think" bs. I'm thinking of Florida right now, but I know the same shit goes on in India, China, countries in Africa, and probably other places too. On a longer scale, as models become more and more advanced, maybe there will be teachers that are constantly absorbing new information and are able to keep students up-to-date with best practices, new research, etc. That would not bode well for the leagues of McKinsey-ites and that whole ilk. I am particularly hopeful that AI will be able to radically advance education. That is, if this hypothetical technology is democratized and kept open, rather than horded and sequestered to the ruling capital class until there exists a financial, educational, biological and altogether insurmountable moat between the ultra-wealthy and the rest of us.
- Taikonerd 4y agoHas anyone used any of the "intelligent tutoring systems" mentioned in this Wikipedia article?[0] Lots of comments here are saying "we'll have an intelligent tutoring system one day, based on LLMs" -- but there are many such systems that already claim to be getting good results now.[1] I'd be interested to hear from anyone who's used one. [0] https://en.wikipedia.org/wiki/Intelligent_tutoring_system https://en.wikipedia.org/wiki/Intelligent_tutoring_system [1] https://www.lesswrong.com/posts/vbWBJGWyWyKyoxLBe/darpa-digital-tutor-four-months-to-total-technical-expertise https://www.lesswrong.com/posts/vbWBJGWyWyKyoxLBe/darpa-digi...
- trane_project 4y agoGoing through the list, it seems like all of them only work on very specialized subjects and/or are not published. I have read about the DARPA tutor, but I don't think it's been made publically available. So I doubt anyone here has used it, unless they were in that specific branch of the military at the time it was used. I don't see how LLMs deliver on this. I think they could speed up how the curriculum is generated, and I have played around with ChatGPT to do just that, but the questions only need to be generated once. It's the guiding of students through the exercises that needs to be personalized, not the exercises themselves. As for their effectiveness, I am the author of Trane (see my other comment). It's worked pretty well for me, but I am splitting my time between my job, working on the software, creating materials, and actual practice, so I can't really say what the upper limits are until I can focus on just the last part.
- admsmz 4y agoI am a customer of mathacademy.com, an intelligent tutoring system with a focus on math. Right now they are putting the finishing touches on their courses like abstract algebra and discrete mathematics. They already have linear algebra and calculus. It absolutely works as advertised and I highly recommend it.
- SunghoYahng 4y agoI took a look at the site you mentioned, but I don't see how it's useful for learning. It uses AI to determine what the learner's current level of knowledge is, but the way the learner learns seems completely typical. Then I may as well read classic textbooks. It doesn't seem to me to be that meaningful to determine the learner's current level of knowledge.
- a-dub 4y agoi wonder how much all the redundant and slightly different resources on the internet today close that gap... this paper is from 1984. also would be curious how good today's llms are at getting someone to grok a topic they're having trouble grasping... (this can be actually a hard problem as it can require understanding the misunderstanding in order to guide away from it effectively.)
- ipnon 4y agoIf I ask ChatGPT how transformers work the results are sufficient but brief. It’s difficult to probe ChatGPT, and it doesn’t have an intuition for your weak points like a tutor does. If I ask ChatGPT about who influenced John Maynard Keynes however it will hallucinate a whole corpus of economists and papers that never existed but whose existence is proposed so confidently it is incredulous. Mainly we should appreciate the dearth of capabilities that arise merely as emergent properties of ever larger language models. Some kind of architecture will be needed for each use case, which means we once again find ourselves facing difficult engineering problems.
- narrator 4y agoThis research will be ignored because if they implemented this individualized learning program with AI, there'd still be a bunch of kids who would do poorly and the smart kids would do great and then people would complain even though all kids would do better. This is kind of like how they got rid of AP classes in California because the racial outcomes were unequal[1]. [1]https://www.nytimes.com/2021/11/04/us/california-math-curriculum-guidelines.html https://www.nytimes.com/2021/11/04/us/california-math-curric... ( https://archive.is/2cHrC https://archive.is/2cHrC )
- beyondCritics 4y agoThat suggests to me we are heading towards coaching with AI teachers in the future.
- credit_guy 4y agoChatGPT kind of agrees with you: While I am a language model trained by OpenAI, I am not a substitute for human teachers and I do not have the ability to solve Bloom's two sigma problem on my own. However, I believe that AI and technology can play a significant role in addressing this problem and improving education outcomes. In the future, AI and personalized learning systems could help to provide individualized feedback and support to students, similar to the benefits of one-on-one tutoring. AI can also be used to provide real-time feedback to teachers, helping them to adjust their teaching methods to better meet the needs of their students. However, it's important to note that while technology can be a valuable tool in education, it cannot replace the human element of teaching entirely. Teachers play a crucial role in creating a supportive learning environment, building relationships with students, and providing guidance and mentorship that goes beyond what a machine can offer. So while AI and technology can certainly help to address Bloom's two sigma problem, they should be seen as complementary tools to support teachers and enhance the learning experience, rather than a replacement for human teachers.
- donfuzius 4y agoWow, I think it's truly amazing that it seems like ChatGPT is the only one in the room mentioning, that «creating a supportive learning environment, building relationships with students» is key to success. I haven't read Bloom's paper in detail, but to me it seems very obvious, that the relationship aspect of 1:1 tutoring is a huge factor in the equation, which (as the AI rightly states), cannot be replaced by better machines.
- ramraj07 4y agoChatGPT is already amazing. I’ve let a smart kid loose on it and he instantly asked these questions: 1. How do you work? 2. What should I learn to understand what you wrote (I am 12)? 3. Write a script for a YouTube video walkthrough of xyz game (can’t remember). The future is gonna be interesting.
- thelock85 4y agoI had a conversation about 10 years ago with a high-ranking DoEd official in Obama’s administration who was gushing over 2 sigma as a strategic framing for public education. Now this person is a very influential funder in edtech and ed reform. I think their heart is in the right place but it’s kinda crazy how no one above the school principal level is actively building strategy or executing tactics to address the basics of a good learning disposition: sleep, nutrition, safety, etc. By the time the robo-tutor is perfected, normalized trauma is going to make it useless for most school kids.
- germinalphrase 4y ago“… it’s kinda crazy how no one above the school principal level is actively building strategy or executing tactics to address the basics of a good learning disposition: sleep, nutrition, safety, etc” By virtue of having been students, many people assume they understand how to be an effective educator - but many problems in schools are not, strictly speaking, education problems; they are social problems they come to school.
- luminati 4y agoAKA the one weird trick to shut up racists and eugenists who think nature is all there is to intelligence.
- dang 4y agoRelated: On Bloom's two sigma problem (2020) - https://news.ycombinator.com/item?id=30438273 https://news.ycombinator.com/item?id=30438273 - Feb 2022 (12 comments) The 2 Sigma Problem (1984) [pdf] - https://news.ycombinator.com/item?id=29312739 https://news.ycombinator.com/item?id=29312739 - Nov 2021 (46 comments) On Bloom's two sigma problem: A systematic review - https://news.ycombinator.com/item?id=24458662 https://news.ycombinator.com/item?id=24458662 - Sept 2020 (10 comments) Bloom's 2 Sigma Problem - https://news.ycombinator.com/item?id=24261345 https://news.ycombinator.com/item?id=24261345 - Aug 2020 (15 comments) Bloom's 2 sigma problem - https://news.ycombinator.com/item?id=18110156 https://news.ycombinator.com/item?id=18110156 - Oct 2018 (18 comments) Solving Bloom's 2 Sigma Problem - https://news.ycombinator.com/item?id=16532846 https://news.ycombinator.com/item?id=16532846 - March 2018 (1 comment)
- alxmng 4y agoIt seems to me the larger issue is most people don’t want to learn, or don’t have the environment conducive to learning. It’s already possible to learn in a self-directed way without a teacher. I didn’t even go to high school, all my higher education happened on my computer using the Internet, and still does. The tools for learning many things without tutors are available. Imagine if we had a perfect tutor using a LLM. From my experience most people won’t take advantage of it anyway. Look at television as a contrived example. Before it was widely available one could have imagined that everyone would be able to learn on video, once the cost fell to where everyone had a TV. But what happened is everyone used the TV to entertain themselves instead.
- jschveibinz 4y agoMastery education is great, and a form of it was shown to be successful in pilot schools throughout the US as early as 1919. The program was called “The Winnetka Plan,” and it was a result of progressive ideas coming from John Dewey. My father attended a high school where the program was used, and he said it was great. He was able to get help from the teacher when he needed it, and he was able to finish his high school requirements a full one year early. This is because mastery is individualized and is based on reaching learning goals not “time in a seat” or “time in grade.” There are many articles about the plan on the Internet. Here is a link to one of them: http://schugurensky.faculty.asu.edu/moments/1919winnetka.html http://schugurensky.faculty.asu.edu/moments/1919winnetka.htm... The question is: why don’t all school systems use this approach? Heck, why don’t more teachers take this approach in their classrooms? If the goal is mastery of certain basic concepts, then why allow some students (sometimes many students) to fail? Unfortunately I don’t know the answer. It could be the fact that students can’t sit still and work independently when they are in a classroom filled with other kids. It could be that schools and teachers are there just to control students rather than teach them. Your guess is as good as mine.
- upwardbound 4y agoIt's just a LOT of work to tailor the lesson plan for 30 students, rather than one. A really really good tutor will customize the day's lesson every day for the child. This is famously envisioned as one of the areas that AI might help with: bringing 1:1 tutoring to the poorest of the world's children. It's the key plot hook of Neal Stephenson's book "The Diamond Age: Or, A Young Lady's Illustrated Primer", wherein said Primer is an AI-powered tablet computer designed to interactively tutor young orphans and other street children to become highly educated.
- teucris 4y agoWhile I loved the idea of the Primer, I think one of the themes of the book was that it actually couldn’t work fully with just AI. All the children who end up “exceptional” (eg not just part of the mouse army) actually had a human tutor behind the scenes.
- anovikov 4y ago"If you throw 30x more resources on a problem, you are usually able to achieve outstanding results even with mediocre quality inputs" - i don't know why it looks like a surprise to anyone.
- maCDzP 4y agoThis is a topic that interest me. I see myself as a life long learner and I want to strive towards mastery. In some topics I’ll maybe get a tutor, it’s expensive but worth it. I’ll share my workflow for studying and I am keen to hear about yours. I start with exercises almost immediately when studying. Practice over theory. I have noticed that an unsolved problem, or a solved problem that I didn’t understand really bugs me. I use that irritation to fuel me when I go over the theory of a subject. When doing exercises I count wrong answers from negligence as worse than just plain wrong. That’s because I believe it could stem from a lack of understanding disguised as negligence. If I am struggling with a concept I’ll first try to copy the chapter of the book by hand. To internalize the author’s voice and framing. If that doesn’t work, I’ll search for another explanation and do the same thing. I prefer books over video lectures. Video is to intense and I can’t as easily set the pace. Some video content out there is amazing, specially with math and visualization of math. That’s really helpful for me. I don’t stress since I am going to this until I die. I play the long game.
- sailorganymede 4y agoThis is one of the most important problems to solve IMO, especially for people who have left the traditional academic system. I feel that having the pressure to perform under examinations, as horrible as it was, at least got some level competency over the line but not having that anymore and learning outside of work has been a tricky affair. I suspect that the solution to this problem might be whatever is common in lots of different learning styles.
- Jinxcosplay 4y ago[dead]
- Cuteaesthetic 4y ago[dead]
- Kawaiiaesthetic 4y ago[dead]
- MeshFresh456 4y ago[dead]
- Veranorooz67 4y ago[dead]
- DorkyDweeb2 4y ago[dead]
- HilariousRed3 4y ago[dead]
- Soverinowu09 4y ago[dead]
- Storystork790 4y ago[dead]
- deleted 4y ago[deleted]