5 ms·

AI used to write homework should be banned. AI in 1:1 tutor mode with proper hardware (live scanning pen and paper), harness and guardrails should be wildly su

by baq 4mo ago

AI used to write homework should be banned. AI in 1:1 tutor mode with proper hardware (live scanning pen and paper), harness and guardrails should be wildly successful (in terms of education outcomes) especially in elementary school.

JumpCrisscross 4mo ago
> AI in 1:1 tutor mode with proper hardware, harness and guardrails should be wildly successful I’m open to the idea! Show me the evidence. Then we can roll it out to our kids.
- baq 4mo ago
  Glad you’re asking. https://cepr.org/publications/dp21577 https://cepr.org/publications/dp21577
  - YawningAngel 4mo ago
    >AI users who maintain similar homework completion time as non-AI users experience small learning losses. Seems like there's no benefit even if it's used "correctly"?
  - JumpCrisscross 4mo ago
    “AI adoption raises homework scores by 18% and reduces completion time by 30%, but lowers monthly exam scores by 20% within six months. High-stakes entrance-exam scores fall by 18 and 24%, with the full penalty emerging only after about two years.” Yup. Short-term metrics juice. Actual comprehension and cognition falls. This seems to be the case across the board, including with adults. I’m genuinely optimistic that there is a way to make AI helpful in education. I just don’t think we’ve found it yet. (We certainly haven’t demonstrated it.)
    - sisve 4mo ago
      > reduces completion time by 30% This is probably the big problem, or at least one of them.. If you use less time on learning, it will probably be harder to remember what you learned also. We need to spend some time to make it stick
      - solid_fuel 4mo ago
        The behavioral issue I see is that LLM users tend to immediately reach for an LLM and do their thinking in concert with it. This tempts users to approach problems by first feeding them into the LLM and then simply following the route the LLM lays out, which does improve task completion time for tasks that the LLM can simply regurgitate, but it stops the user from developing the actual critical thinking skills that school is supposed to teach.
        sisve 4mo ago
        Can agree on that. The description of the paper also said: AI users who maintain similar homework completion time as non-AI users experience small learning losses. This was a surprise too me. I would have thought otherwise. Would love to see some evidence about if more or less people fall behind and have worse results. In my head the AI should be able to get the weakest students a bit highere.
        JumpCrisscross 4mo ago
        > In my head the AI should be able to get the weakest students a bit higher I think the evidence so far is all students lose learning and cognition, but the brighter students lose less.
        sisve 4mo ago
        Based on the paper it hits harder on the brighter From https://cepr.org/publications/dp21577 https://cepr.org/publications/dp21577 : The losses are largest in social science subjects, followed by STEM and languages, and are especially large for junior students, high-achieving students, and boys. No mention of the weakest student. Which probably means they did not have a significant worse or better score
    - ApolloFortyNine 4mo ago
      I think AI could (and by some students probably already is) be used to help a student better understand the material, and faster than you could before. I still recall some parts of Physics taking a while to click, and often having to reread different sections of a textbook to try and understand the what and why behind something. The biggest issue is a child has to want to do that, since they also could just ask the AI for the answer and then go back to playing video games. End of the day past age 13 or so I just don't see legislation making any difference, they'll find a way past any law blocking them from using AI. Like a lot of education it'll probably come down to parenting.
      - JumpCrisscross 4mo ago
        > I think AI could (and by some students probably already is) be used to help a student better understand the material, and faster than you could before I think so too. But we haven’t demonstrated we’ve found how, in kids or in adults. > biggest issue is We genuinely don’t know what the biggest issue is. We just know it doesn’t work. There is zero quality evidence for AI helping with learning or cognition in kids or adults. (Happy to be proven wrong. This is a fast-moving and big field.) > they'll find a way past any law blocking them from using AI. Like a lot of education it'll probably come down to parenting And community. Rich towns restrict devices in school, monitor use at home and thus will have less of an issue with AI exposure.
        ApolloFortyNine 4mo ago
        >I think so too. But we haven’t demonstrated we’ve found how, in kids or in adults. Ask chatgpt or claude, on their highest model (probably unnecessary but I'm sensing a vibe) to explain a simple linear algebra problem, and if you don't understand it, ask about what part you don't understand. And if you truly believe it made something up, prove it. This is seriously the easiest thing to prove out there, you can see for yourself in the next 5 minutes.
        JumpCrisscross 4mo ago
        Sure. I already know linear algebra. If it’s a new branch of mathematics, this is a terrible way to learn. Again, the research points almost exclusively in one direction when it comes to learning and cognition around AI. You’ll solve more problems more quickly but wind up learning and thinking less.
    - oneshtein 4mo ago
      With AI — +18%, without AI — -20%. Conclusion?
  - lelandfe 4mo ago
    Care to give us the bits you found interesting in the paper to spare me plonking down £6? Would hate to dissect this just off a paragraph.
    - solid_fuel 4mo ago
      Considering that the paper concludes that even students who take the long approach and use LLMs in the most appropriate way for learning still retain less over the long term than students who simply don't use LLMs, I think it's likely they didn't read the paper in the first place.
forgetfreeman 4mo ago
We thought the same of electronic devices in general and digital learning content specifically. In actual practice both result in lowered test scores and declining critical thinking skills.
SubmarineClub 4mo ago
Idk why you screeching AI touts are so confident about its ‘wild’ success in all areas given absolutely zero evidence to that effect. It’s tiresome.
- LtWorf 4mo ago
  It's inevitably your fault for prompting incorrectly or using the wrong model.
  - solid_fuel 4mo ago
    "You just have to repeat the prompt 3 times and then spin around counter-clockwise twice! That always works for me. You obviously just don't know how to prompt the model correctly." Every time I see LLM enjoyers yapping on like this, it just reminds me of people trying to read tea leaves. There's all these goofy little rules about how to structure the prompt and how mean or nice to be to get it to work optimally, but I think it's obvious that most of these users are just seeing incidental successful outcomes in a largely random system and extrapolating from there because it makes them feel in control. It is, quite literally, superstition.
    - kklisura 4mo ago
      Instead of prompts, let’s call them incantations.
solid_fuel 4mo ago
Disagree. AI has no business being used in 1:1 tutor mode before the hallucination and sycophancy issues are completely resolved. As is, I can easily see it being a hindrance to building actual understanding. Just one example - it's very common to see ChatGPT and the like respond with "you're absolutely correct! Great insight" to something that is a complete misunderstanding.
- JumpCrisscross 4mo ago
  Just realized 1:1 AI is 90s self-esteem medals-for-everyone parenting on steroids.
- therealdrag0 4mo ago
  Teachers hallucinate too. I’ve had creationists and communists and tin-foil-hat (chem trails, 5g, etc) teachers. Surely you can imagine an AI tutor that is higher than zero ROI.
  - solid_fuel 4mo ago
    > I’ve had creationists and communists and tin-foil-hat (chem trails, 5g, etc) teachers. I certainly have, too, but there is still a difference between a person who has a factually incorrect but consistent worldview and an LLM which simply reflects the worldview of the user or even changes between queries. I don't think creationists have any business being in schools either, for what it's worth, but I think it's easier for a teenager to sort out "Mr. Smith has no clue what he's talking about" vs "I have no clue what's true because the LLM everyone expects me to learn from just confirms everything I ask regardless of what I'm asking".
  - beejiu 4mo ago
    A bit part of education is (should be) independent learning with textbooks and reading. You don't need to be "tutored".
    - HKH2 4mo ago
      Tutors should be able to approximate the ZPD better than any student can. Most students lack intrinsic motivation and it's a tutor's job to help them get started.
wolvoleo 4mo ago
I think AI should be used in higher level schools but with the added requirement that the output will be held to a much higher standard and that it's fact checked. Teach the students to use AI to reach a higher level while mitigating the inherent issues like hallucination and sycophancy.
logickkk1 4mo ago
fwiw, Alpha School is the supervised version. the New York campus is $65k/yr and not legally a school. private school money with homeschool paperwork and an app doing the teaching. https://www.wired.com/story/alpha-schools-new-york-city-campus-isnt-actually-a-school/ https://www.wired.com/story/alpha-schools-new-york-city-camp...
asdff 4mo ago
Tutor mode sucks because even if it was actually accurate and didn't hallucinate, it isn't scoped to what the class is actually covering within a given topic. Best AI is still your own brain, trained on paying attention in class and reading the assigned content.
- graemep 4mo ago
  One to one human tutors are the best way of educating kids and they are not scoped to what a class is covering. They teach the kid as an individual: they are effective because they can adjust to the kids abilities and interests.
  - asdff 4mo ago
    It would be a shitty tutor if they never asked to see your textbook and what sort of material you'd work on. You'd legitimately waste time covering stuff that won't ever be tested.

by baq 4mo ago

JumpCrisscross 4mo ago

> AI in 1:1 tutor mode with proper hardware, harness and guardrails should be wildly successful I’m open to the idea! Show me the evidence. Then we can roll it out to our kids.

baq 4mo ago
Glad you’re asking. https://cepr.org/publications/dp21577 https://cepr.org/publications/dp21577
- YawningAngel 4mo ago
  >AI users who maintain similar homework completion time as non-AI users experience small learning losses. Seems like there's no benefit even if it's used "correctly"?
- JumpCrisscross 4mo ago
  “AI adoption raises homework scores by 18% and reduces completion time by 30%, but lowers monthly exam scores by 20% within six months. High-stakes entrance-exam scores fall by 18 and 24%, with the full penalty emerging only after about two years.” Yup. Short-term metrics juice. Actual comprehension and cognition falls. This seems to be the case across the board, including with adults. I’m genuinely optimistic that there is a way to make AI helpful in education. I just don’t think we’ve found it yet. (We certainly haven’t demonstrated it.)
  - sisve 4mo ago
    > reduces completion time by 30% This is probably the big problem, or at least one of them.. If you use less time on learning, it will probably be harder to remember what you learned also. We need to spend some time to make it stick
    - solid_fuel 4mo ago
      The behavioral issue I see is that LLM users tend to immediately reach for an LLM and do their thinking in concert with it. This tempts users to approach problems by first feeding them into the LLM and then simply following the route the LLM lays out, which does improve task completion time for tasks that the LLM can simply regurgitate, but it stops the user from developing the actual critical thinking skills that school is supposed to teach.
      - sisve 4mo ago
        Can agree on that. The description of the paper also said: AI users who maintain similar homework completion time as non-AI users experience small learning losses. This was a surprise too me. I would have thought otherwise. Would love to see some evidence about if more or less people fall behind and have worse results. In my head the AI should be able to get the weakest students a bit highere.
        JumpCrisscross 4mo ago
        > In my head the AI should be able to get the weakest students a bit higher I think the evidence so far is all students lose learning and cognition, but the brighter students lose less.
        sisve 4mo ago
        Based on the paper it hits harder on the brighter From https://cepr.org/publications/dp21577 https://cepr.org/publications/dp21577 : The losses are largest in social science subjects, followed by STEM and languages, and are especially large for junior students, high-achieving students, and boys. No mention of the weakest student. Which probably means they did not have a significant worse or better score
  - ApolloFortyNine 4mo ago
    I think AI could (and by some students probably already is) be used to help a student better understand the material, and faster than you could before. I still recall some parts of Physics taking a while to click, and often having to reread different sections of a textbook to try and understand the what and why behind something. The biggest issue is a child has to want to do that, since they also could just ask the AI for the answer and then go back to playing video games. End of the day past age 13 or so I just don't see legislation making any difference, they'll find a way past any law blocking them from using AI. Like a lot of education it'll probably come down to parenting.
    - JumpCrisscross 4mo ago
      > I think AI could (and by some students probably already is) be used to help a student better understand the material, and faster than you could before I think so too. But we haven’t demonstrated we’ve found how, in kids or in adults. > biggest issue is We genuinely don’t know what the biggest issue is. We just know it doesn’t work. There is zero quality evidence for AI helping with learning or cognition in kids or adults. (Happy to be proven wrong. This is a fast-moving and big field.) > they'll find a way past any law blocking them from using AI. Like a lot of education it'll probably come down to parenting And community. Rich towns restrict devices in school, monitor use at home and thus will have less of an issue with AI exposure.
      - ApolloFortyNine 4mo ago
        >I think so too. But we haven’t demonstrated we’ve found how, in kids or in adults. Ask chatgpt or claude, on their highest model (probably unnecessary but I'm sensing a vibe) to explain a simple linear algebra problem, and if you don't understand it, ask about what part you don't understand. And if you truly believe it made something up, prove it. This is seriously the easiest thing to prove out there, you can see for yourself in the next 5 minutes.
        JumpCrisscross 4mo ago
        Sure. I already know linear algebra. If it’s a new branch of mathematics, this is a terrible way to learn. Again, the research points almost exclusively in one direction when it comes to learning and cognition around AI. You’ll solve more problems more quickly but wind up learning and thinking less.
  - oneshtein 4mo ago
    With AI — +18%, without AI — -20%. Conclusion?
- lelandfe 4mo ago
  Care to give us the bits you found interesting in the paper to spare me plonking down £6? Would hate to dissect this just off a paragraph.
  - solid_fuel 4mo ago
    Considering that the paper concludes that even students who take the long approach and use LLMs in the most appropriate way for learning still retain less over the long term than students who simply don't use LLMs, I think it's likely they didn't read the paper in the first place.

forgetfreeman 4mo ago

We thought the same of electronic devices in general and digital learning content specifically. In actual practice both result in lowered test scores and declining critical thinking skills.

SubmarineClub 4mo ago

Idk why you screeching AI touts are so confident about its ‘wild’ success in all areas given absolutely zero evidence to that effect. It’s tiresome.

LtWorf 4mo ago
It's inevitably your fault for prompting incorrectly or using the wrong model.
- solid_fuel 4mo ago
  "You just have to repeat the prompt 3 times and then spin around counter-clockwise twice! That always works for me. You obviously just don't know how to prompt the model correctly." Every time I see LLM enjoyers yapping on like this, it just reminds me of people trying to read tea leaves. There's all these goofy little rules about how to structure the prompt and how mean or nice to be to get it to work optimally, but I think it's obvious that most of these users are just seeing incidental successful outcomes in a largely random system and extrapolating from there because it makes them feel in control. It is, quite literally, superstition.
  - kklisura 4mo ago
    Instead of prompts, let’s call them incantations.

solid_fuel 4mo ago

Disagree. AI has no business being used in 1:1 tutor mode before the hallucination and sycophancy issues are completely resolved. As is, I can easily see it being a hindrance to building actual understanding. Just one example - it's very common to see ChatGPT and the like respond with "you're absolutely correct! Great insight" to something that is a complete misunderstanding.

JumpCrisscross 4mo ago
Just realized 1:1 AI is 90s self-esteem medals-for-everyone parenting on steroids.
therealdrag0 4mo ago
Teachers hallucinate too. I’ve had creationists and communists and tin-foil-hat (chem trails, 5g, etc) teachers. Surely you can imagine an AI tutor that is higher than zero ROI.
- solid_fuel 4mo ago
  > I’ve had creationists and communists and tin-foil-hat (chem trails, 5g, etc) teachers. I certainly have, too, but there is still a difference between a person who has a factually incorrect but consistent worldview and an LLM which simply reflects the worldview of the user or even changes between queries. I don't think creationists have any business being in schools either, for what it's worth, but I think it's easier for a teenager to sort out "Mr. Smith has no clue what he's talking about" vs "I have no clue what's true because the LLM everyone expects me to learn from just confirms everything I ask regardless of what I'm asking".
- beejiu 4mo ago
  A bit part of education is (should be) independent learning with textbooks and reading. You don't need to be "tutored".
  - HKH2 4mo ago
    Tutors should be able to approximate the ZPD better than any student can. Most students lack intrinsic motivation and it's a tutor's job to help them get started.

wolvoleo 4mo ago

I think AI should be used in higher level schools but with the added requirement that the output will be held to a much higher standard and that it's fact checked. Teach the students to use AI to reach a higher level while mitigating the inherent issues like hallucination and sycophancy.

logickkk1 4mo ago

fwiw, Alpha School is the supervised version. the New York campus is $65k/yr and not legally a school. private school money with homeschool paperwork and an app doing the teaching. https://www.wired.com/story/alpha-schools-new-york-city-campus-isnt-actually-a-school/ https://www.wired.com/story/alpha-schools-new-york-city-camp...

asdff 4mo ago

Tutor mode sucks because even if it was actually accurate and didn't hallucinate, it isn't scoped to what the class is actually covering within a given topic. Best AI is still your own brain, trained on paying attention in class and reading the assigned content.

graemep 4mo ago
One to one human tutors are the best way of educating kids and they are not scoped to what a class is covering. They teach the kid as an individual: they are effective because they can adjust to the kids abilities and interests.
- asdff 4mo ago
  It would be a shitty tutor if they never asked to see your textbook and what sort of material you'd work on. You'd legitimately waste time covering stuff that won't ever be tested.