5 ms·

I wonder if this kind of intelligent tutoring could be the answer to Bloom's Two Sigma Problem. The limiting factor with that problem was that not everyone can

I wonder if this kind of intelligent tutoring could be the answer to Bloom's Two Sigma Problem. The limiting factor with that problem was that not everyone can afford a personal tutor. Having an AI tutor that can breeze through the SAT seems like it should give every student a major boost.

alach11 4y ago
My mind went to the same place. Knowing that "the average tutored student was above 98% of the students in the control class", imagine how transformative it will be to lift the average student to the 98th percentile! This may be the most transformative change to education since the invention of public schooling.
- sophiabits 4y ago
  Every now and then I think about Marc Andreessen’s “software is eating the world” article. He was extremely excited about investing in EdTech and HealthTech software companies at the time—for obvious reasons! But EdTech hasn’t really delivered much value. The modern LMS feels to me like a reinvention of Facebook groups in many ways… LLMs feel like something that _could_ very well end up eating education, and that’s a really exciting prospect. My personal experience with schooling was less than stellar, and figuring out alternative and more scalable ways to get knowledge into kids’ heads has a lot of possible upside for society imo
  - falcor84 4y ago
    > The modern LMS feels to me like a reinvention of Facebook groups in many ways I agree with your sentiment that there's a lot more potential value to be derived from EdTech than we've seen so far, but even if LMS's are "just" like Facebook groups, that's already the equivalent of hundreds of billions of dollars in economic value. When I was a kid, if I was sick for a few days, I had to manually copy my friend's notes after I returned to school, and would still feel confused. Now, with an LMS, I have access to all the learning content from home, for my own class, and for resources from similar classes taught by amazing teachers from across the world, at any modality I prefer. This on its own is incredible value. (although with the caveat that being sick is no longer the excuse to rest and relax as it used to be)
james-revisoai 4y ago
Since using T5 to generate quiz questions in 2019 I've thought this. Transformers now can understand misconceptions, quote source materials... it will equalise education.
oh_sigh 4y ago
Every student might get an absolute boost, but the relative gap may even widen. Consider students who can afford to use cutting edge, expensive to train/run models, vs students who can only use off the shelf, older models.
- visarga 4y ago
  More likely AI will be cheap and widespread like electricity. Data likes to be free. Assuming the divide will be real, you can just record a million problem solutions from ExpensiveLM and use it to tune your CheapLM, it will be a specialised cheap model that works ok on your task of interest.
dopeboy 4y ago
Circa late 2015, early 2016, my cofounder and I spent ~6 month attempting to build exactly this. Our hypothesis was that students self studying for the SAT used books (eg Kaplan) but the solutions in the back were poor in explaining how they arrived to the solution because they usually just cut to the chase rather than explaining the approach & helping you arrive to the solution. So we took each practice SAT math question and wrote out a solution spread over four swipeable pages. Our hope was that if you saw the first or second page, you'd arrive at the solution without having to see the answer, thus increasing your chance to solve it on your own next time. We spent over 100 hours tutoring kids and had them use our MVP to mild success. The obvious next step was to automate the solutions rather than hand write it ourselves - like GPT-4 has done here. As someone who has logged over 2000 hours teaching, there's a lot of soft nuances and tactics to tutoring. Also, motivation is a big thing a good tutor brings. I don't see this solution as quite there yet but getting close. Very optimistic here.
- terribleperson 4y ago
  The weird thing is how rarely approaches like the one you describe here are applied. It's not exactly a secret that simply providing the answer to math questions is great if someone gets them all correct, but nearly useless when someone gets them wrong. It's a problem that's been solved before - in a classroom, the traditional answer is working through the problem on the board and hopefully students are following along. Accompanying books with videos that work through the problems has been done at least since we had VHS. So why are there still so many bad study books?
tgv 4y ago
It's not the tutoring alone. It's also the social effect. A machine simply does not exert that kind of influence, nor do online courses require the same amount of dedication and ambition. Compare MOOCs: they have incredibly high drop-out rates.
- asdff 4y ago
  Chances are if you have a tutor you are paying good money for it, and have made appointments at regular intervals, that probably helps with motivation too. I think its possible that chatchpt and other learning ai will end up being like how office hours are used today in college. Like office hours, these tools are available for everyone, but for one reason or another not everyone will bother with them or use them to their fullest.
recuter 4y ago
A tutor doesn't just imply parents who can afford one but also who care enough to do so in the first place and see the value of an education. I think you'd find that even by the sat-prep age the actual limiting factor for many (most?) of the tutor-less is a fate-accompli attitude towards their path in life. Might as well turn on the chat bots at the cradle. Ah, and make sure everybody gets the same quality AI to parent them for it to be egalitarian.
- alach11 4y ago
  Right... but the whole point Bloom's experiment is that if you give every kid a tutor, they perform (on average) at the 98th percentile. Not just the rich kids who can afford tutors. I do think there are valid concerns that this technology (like old-school human tutoring) could be disproportionately available to already-privileged children. This could potentially widen the achievement gap. But it seems way more likely that we can get poor kids a laptop to tutor them vs. getting them a dedicated human tutor.
  - asdff 4y ago
    Blooms experiment would probably more relevant here if he gave the kids the option of using a tutor, at their own liberty and pacing (not sure if he did in fact do so). Everyone has chatgpt here but not everyone will use it. Just like how not everyone shows up to office hours or asks questions of their teacher.
asdff 4y ago
On the other hand, I think it could also trigger an arms race, and might very well incentivize learning how to write a good prompt more than learning the concept well enough to succeed in a class. This reminds me in my era of the divide in success in my era, between students who were able to navigate the internet to the benefit of their academic success, and those that weren't able to do things like quickly paraphrase a wikipedia article on the subject and use the wikipedia citations as their own, find a relevant flashcard set someone else made and shared on quizlet, or even find pdfs of the teacher's solutions accompanying the textbook. Cheat codes are cheat codes. They will help in terms of your grade but usually hurt in terms of understanding.
- mullingitover 4y ago
  > On the other hand, I think it could also trigger an arms race, and might very well incentivize learning how to write a good prompt more than learning the concept well enough to succeed in a class. I think general public having access to these AI tools is going to create a shift from scholastic aptitude testing to more performance testing. So no more homework, you use your time out of class to train so you can use the in-class time to perform based on the training you've been doing with your personal AI tutor.

alach11 4y ago

My mind went to the same place. Knowing that "the average tutored student was above 98% of the students in the control class", imagine how transformative it will be to lift the average student to the 98th percentile! This may be the most transformative change to education since the invention of public schooling.

sophiabits 4y ago
Every now and then I think about Marc Andreessen’s “software is eating the world” article. He was extremely excited about investing in EdTech and HealthTech software companies at the time—for obvious reasons! But EdTech hasn’t really delivered much value. The modern LMS feels to me like a reinvention of Facebook groups in many ways… LLMs feel like something that _could_ very well end up eating education, and that’s a really exciting prospect. My personal experience with schooling was less than stellar, and figuring out alternative and more scalable ways to get knowledge into kids’ heads has a lot of possible upside for society imo
- falcor84 4y ago
  > The modern LMS feels to me like a reinvention of Facebook groups in many ways I agree with your sentiment that there's a lot more potential value to be derived from EdTech than we've seen so far, but even if LMS's are "just" like Facebook groups, that's already the equivalent of hundreds of billions of dollars in economic value. When I was a kid, if I was sick for a few days, I had to manually copy my friend's notes after I returned to school, and would still feel confused. Now, with an LMS, I have access to all the learning content from home, for my own class, and for resources from similar classes taught by amazing teachers from across the world, at any modality I prefer. This on its own is incredible value. (although with the caveat that being sick is no longer the excuse to rest and relax as it used to be)

james-revisoai 4y ago

Since using T5 to generate quiz questions in 2019 I've thought this. Transformers now can understand misconceptions, quote source materials... it will equalise education.

oh_sigh 4y ago

Every student might get an absolute boost, but the relative gap may even widen. Consider students who can afford to use cutting edge, expensive to train/run models, vs students who can only use off the shelf, older models.

visarga 4y ago
More likely AI will be cheap and widespread like electricity. Data likes to be free. Assuming the divide will be real, you can just record a million problem solutions from ExpensiveLM and use it to tune your CheapLM, it will be a specialised cheap model that works ok on your task of interest.

dopeboy 4y ago

Circa late 2015, early 2016, my cofounder and I spent ~6 month attempting to build exactly this. Our hypothesis was that students self studying for the SAT used books (eg Kaplan) but the solutions in the back were poor in explaining how they arrived to the solution because they usually just cut to the chase rather than explaining the approach & helping you arrive to the solution. So we took each practice SAT math question and wrote out a solution spread over four swipeable pages. Our hope was that if you saw the first or second page, you'd arrive at the solution without having to see the answer, thus increasing your chance to solve it on your own next time. We spent over 100 hours tutoring kids and had them use our MVP to mild success. The obvious next step was to automate the solutions rather than hand write it ourselves - like GPT-4 has done here. As someone who has logged over 2000 hours teaching, there's a lot of soft nuances and tactics to tutoring. Also, motivation is a big thing a good tutor brings. I don't see this solution as quite there yet but getting close. Very optimistic here.

terribleperson 4y ago
The weird thing is how rarely approaches like the one you describe here are applied. It's not exactly a secret that simply providing the answer to math questions is great if someone gets them all correct, but nearly useless when someone gets them wrong. It's a problem that's been solved before - in a classroom, the traditional answer is working through the problem on the board and hopefully students are following along. Accompanying books with videos that work through the problems has been done at least since we had VHS. So why are there still so many bad study books?

tgv 4y ago

It's not the tutoring alone. It's also the social effect. A machine simply does not exert that kind of influence, nor do online courses require the same amount of dedication and ambition. Compare MOOCs: they have incredibly high drop-out rates.

asdff 4y ago
Chances are if you have a tutor you are paying good money for it, and have made appointments at regular intervals, that probably helps with motivation too. I think its possible that chatchpt and other learning ai will end up being like how office hours are used today in college. Like office hours, these tools are available for everyone, but for one reason or another not everyone will bother with them or use them to their fullest.

recuter 4y ago

A tutor doesn't just imply parents who can afford one but also who care enough to do so in the first place and see the value of an education. I think you'd find that even by the sat-prep age the actual limiting factor for many (most?) of the tutor-less is a fate-accompli attitude towards their path in life. Might as well turn on the chat bots at the cradle. Ah, and make sure everybody gets the same quality AI to parent them for it to be egalitarian.

alach11 4y ago
Right... but the whole point Bloom's experiment is that if you give every kid a tutor, they perform (on average) at the 98th percentile. Not just the rich kids who can afford tutors. I do think there are valid concerns that this technology (like old-school human tutoring) could be disproportionately available to already-privileged children. This could potentially widen the achievement gap. But it seems way more likely that we can get poor kids a laptop to tutor them vs. getting them a dedicated human tutor.
- asdff 4y ago
  Blooms experiment would probably more relevant here if he gave the kids the option of using a tutor, at their own liberty and pacing (not sure if he did in fact do so). Everyone has chatgpt here but not everyone will use it. Just like how not everyone shows up to office hours or asks questions of their teacher.

asdff 4y ago

On the other hand, I think it could also trigger an arms race, and might very well incentivize learning how to write a good prompt more than learning the concept well enough to succeed in a class. This reminds me in my era of the divide in success in my era, between students who were able to navigate the internet to the benefit of their academic success, and those that weren't able to do things like quickly paraphrase a wikipedia article on the subject and use the wikipedia citations as their own, find a relevant flashcard set someone else made and shared on quizlet, or even find pdfs of the teacher's solutions accompanying the textbook. Cheat codes are cheat codes. They will help in terms of your grade but usually hurt in terms of understanding.

mullingitover 4y ago
> On the other hand, I think it could also trigger an arms race, and might very well incentivize learning how to write a good prompt more than learning the concept well enough to succeed in a class. I think general public having access to these AI tools is going to create a shift from scholastic aptitude testing to more performance testing. So no more homework, you use your time out of class to train so you can use the in-class time to perform based on the training you've been doing with your personal AI tutor.