10 ms·

Glad you’re asking. https://cepr.org/publications/dp21577 https://cepr.org/publications/dp21577

by baq 4mo ago

YawningAngel 4mo ago
>AI users who maintain similar homework completion time as non-AI users experience small learning losses. Seems like there's no benefit even if it's used "correctly"?
JumpCrisscross 4mo ago
“AI adoption raises homework scores by 18% and reduces completion time by 30%, but lowers monthly exam scores by 20% within six months. High-stakes entrance-exam scores fall by 18 and 24%, with the full penalty emerging only after about two years.” Yup. Short-term metrics juice. Actual comprehension and cognition falls. This seems to be the case across the board, including with adults. I’m genuinely optimistic that there is a way to make AI helpful in education. I just don’t think we’ve found it yet. (We certainly haven’t demonstrated it.)
- sisve 4mo ago
  > reduces completion time by 30% This is probably the big problem, or at least one of them.. If you use less time on learning, it will probably be harder to remember what you learned also. We need to spend some time to make it stick
  - solid_fuel 4mo ago
    The behavioral issue I see is that LLM users tend to immediately reach for an LLM and do their thinking in concert with it. This tempts users to approach problems by first feeding them into the LLM and then simply following the route the LLM lays out, which does improve task completion time for tasks that the LLM can simply regurgitate, but it stops the user from developing the actual critical thinking skills that school is supposed to teach.
    - sisve 4mo ago
      Can agree on that. The description of the paper also said: AI users who maintain similar homework completion time as non-AI users experience small learning losses. This was a surprise too me. I would have thought otherwise. Would love to see some evidence about if more or less people fall behind and have worse results. In my head the AI should be able to get the weakest students a bit highere.
      - JumpCrisscross 4mo ago
        > In my head the AI should be able to get the weakest students a bit higher I think the evidence so far is all students lose learning and cognition, but the brighter students lose less.
        sisve 4mo ago
        Based on the paper it hits harder on the brighter From https://cepr.org/publications/dp21577 https://cepr.org/publications/dp21577 : The losses are largest in social science subjects, followed by STEM and languages, and are especially large for junior students, high-achieving students, and boys. No mention of the weakest student. Which probably means they did not have a significant worse or better score
    - cheesecakegood 4mo ago
      It’s not just critical thinking skills, it’s also that there’s a big difference between recognition/following instructions, and recall/generating your own memories of an approach. But most students don’t recognize the difference. In other words, “following the route” is a big part of the problem - it doesn’t engage the brain the same and isn’t representative of real world use, and having something explained well doesn’t mean you can in turn explain it well yourself (the more revealing test of internalized true understanding)
    - scotty79 4mo ago
      > reach for an LLM and do their thinking in concert with it This basically models how the intellectual work is going to be overwhelmingly done in the future.
- ApolloFortyNine 4mo ago
  I think AI could (and by some students probably already is) be used to help a student better understand the material, and faster than you could before. I still recall some parts of Physics taking a while to click, and often having to reread different sections of a textbook to try and understand the what and why behind something. The biggest issue is a child has to want to do that, since they also could just ask the AI for the answer and then go back to playing video games. End of the day past age 13 or so I just don't see legislation making any difference, they'll find a way past any law blocking them from using AI. Like a lot of education it'll probably come down to parenting.
  - JumpCrisscross 4mo ago
    > I think AI could (and by some students probably already is) be used to help a student better understand the material, and faster than you could before I think so too. But we haven’t demonstrated we’ve found how, in kids or in adults. > biggest issue is We genuinely don’t know what the biggest issue is. We just know it doesn’t work. There is zero quality evidence for AI helping with learning or cognition in kids or adults. (Happy to be proven wrong. This is a fast-moving and big field.) > they'll find a way past any law blocking them from using AI. Like a lot of education it'll probably come down to parenting And community. Rich towns restrict devices in school, monitor use at home and thus will have less of an issue with AI exposure.
    - ApolloFortyNine 4mo ago
      >I think so too. But we haven’t demonstrated we’ve found how, in kids or in adults. Ask chatgpt or claude, on their highest model (probably unnecessary but I'm sensing a vibe) to explain a simple linear algebra problem, and if you don't understand it, ask about what part you don't understand. And if you truly believe it made something up, prove it. This is seriously the easiest thing to prove out there, you can see for yourself in the next 5 minutes.
      - JumpCrisscross 4mo ago
        Sure. I already know linear algebra. If it’s a new branch of mathematics, this is a terrible way to learn. Again, the research points almost exclusively in one direction when it comes to learning and cognition around AI. You’ll solve more problems more quickly but wind up learning and thinking less.
- oneshtein 4mo ago
  With AI — +18%, without AI — -20%. Conclusion?
lelandfe 4mo ago
Care to give us the bits you found interesting in the paper to spare me plonking down £6? Would hate to dissect this just off a paragraph.
- solid_fuel 4mo ago
  Considering that the paper concludes that even students who take the long approach and use LLMs in the most appropriate way for learning still retain less over the long term than students who simply don't use LLMs, I think it's likely they didn't read the paper in the first place.

Glad you’re asking. https://cepr.org/publications/dp21577 https://cepr.org/publications/dp21577

by baq 4mo ago

Glad you’re asking. https://cepr.org/publications/dp21577 https://cepr.org/publications/dp21577

>AI users who maintain similar homework completion time as non-AI users experience small learning losses. Seems like there's no benefit even if it's used "correctly"?

JumpCrisscross 4mo ago

“AI adoption raises homework scores by 18% and reduces completion time by 30%, but lowers monthly exam scores by 20% within six months. High-stakes entrance-exam scores fall by 18 and 24%, with the full penalty emerging only after about two years.” Yup. Short-term metrics juice. Actual comprehension and cognition falls. This seems to be the case across the board, including with adults. I’m genuinely optimistic that there is a way to make AI helpful in education. I just don’t think we’ve found it yet. (We certainly haven’t demonstrated it.)

sisve 4mo ago
> reduces completion time by 30% This is probably the big problem, or at least one of them.. If you use less time on learning, it will probably be harder to remember what you learned also. We need to spend some time to make it stick
- solid_fuel 4mo ago
  The behavioral issue I see is that LLM users tend to immediately reach for an LLM and do their thinking in concert with it. This tempts users to approach problems by first feeding them into the LLM and then simply following the route the LLM lays out, which does improve task completion time for tasks that the LLM can simply regurgitate, but it stops the user from developing the actual critical thinking skills that school is supposed to teach.
  - sisve 4mo ago
    Can agree on that. The description of the paper also said: AI users who maintain similar homework completion time as non-AI users experience small learning losses. This was a surprise too me. I would have thought otherwise. Would love to see some evidence about if more or less people fall behind and have worse results. In my head the AI should be able to get the weakest students a bit highere.
    - JumpCrisscross 4mo ago
      > In my head the AI should be able to get the weakest students a bit higher I think the evidence so far is all students lose learning and cognition, but the brighter students lose less.
      - sisve 4mo ago
        Based on the paper it hits harder on the brighter From https://cepr.org/publications/dp21577 https://cepr.org/publications/dp21577 : The losses are largest in social science subjects, followed by STEM and languages, and are especially large for junior students, high-achieving students, and boys. No mention of the weakest student. Which probably means they did not have a significant worse or better score
  - cheesecakegood 4mo ago
    It’s not just critical thinking skills, it’s also that there’s a big difference between recognition/following instructions, and recall/generating your own memories of an approach. But most students don’t recognize the difference. In other words, “following the route” is a big part of the problem - it doesn’t engage the brain the same and isn’t representative of real world use, and having something explained well doesn’t mean you can in turn explain it well yourself (the more revealing test of internalized true understanding)
  - scotty79 4mo ago
    > reach for an LLM and do their thinking in concert with it This basically models how the intellectual work is going to be overwhelmingly done in the future.
ApolloFortyNine 4mo ago
I think AI could (and by some students probably already is) be used to help a student better understand the material, and faster than you could before. I still recall some parts of Physics taking a while to click, and often having to reread different sections of a textbook to try and understand the what and why behind something. The biggest issue is a child has to want to do that, since they also could just ask the AI for the answer and then go back to playing video games. End of the day past age 13 or so I just don't see legislation making any difference, they'll find a way past any law blocking them from using AI. Like a lot of education it'll probably come down to parenting.
- JumpCrisscross 4mo ago
  > I think AI could (and by some students probably already is) be used to help a student better understand the material, and faster than you could before I think so too. But we haven’t demonstrated we’ve found how, in kids or in adults. > biggest issue is We genuinely don’t know what the biggest issue is. We just know it doesn’t work. There is zero quality evidence for AI helping with learning or cognition in kids or adults. (Happy to be proven wrong. This is a fast-moving and big field.) > they'll find a way past any law blocking them from using AI. Like a lot of education it'll probably come down to parenting And community. Rich towns restrict devices in school, monitor use at home and thus will have less of an issue with AI exposure.
  - ApolloFortyNine 4mo ago
    >I think so too. But we haven’t demonstrated we’ve found how, in kids or in adults. Ask chatgpt or claude, on their highest model (probably unnecessary but I'm sensing a vibe) to explain a simple linear algebra problem, and if you don't understand it, ask about what part you don't understand. And if you truly believe it made something up, prove it. This is seriously the easiest thing to prove out there, you can see for yourself in the next 5 minutes.
    - JumpCrisscross 4mo ago
      Sure. I already know linear algebra. If it’s a new branch of mathematics, this is a terrible way to learn. Again, the research points almost exclusively in one direction when it comes to learning and cognition around AI. You’ll solve more problems more quickly but wind up learning and thinking less.
oneshtein 4mo ago
With AI — +18%, without AI — -20%. Conclusion?

lelandfe 4mo ago

Care to give us the bits you found interesting in the paper to spare me plonking down £6? Would hate to dissect this just off a paragraph.

solid_fuel 4mo ago
Considering that the paper concludes that even students who take the long approach and use LLMs in the most appropriate way for learning still retain less over the long term than students who simply don't use LLMs, I think it's likely they didn't read the paper in the first place.