5 ms·

It's not so clear that AI is an amplifier.. The paper has some fascinating analysis on this topic: "At the other end of the distribution, AI students who spend

by kzz102 1mo ago

It's not so clear that AI is an amplifier.. The paper has some fascinating analysis on this topic: "At the other end of the distribution, AI students who spend more than 65 minutes on their homework receive homework and exam scores similar to those of non-AI students, suggesting that these students do not use generative AI for homework assignments. However, this group consists entirely of students who adopted generative AI no more than Öve months. Six months after adoption, no AI student spends more than 65 minutes completing their homework (see Figure A5). This is consistent with the gradual process of learning how to use AI tools. It also suggests that AI crowds out the highest level of e§ort." "Interestingly, in the range of 50-65 minutes, the median and the interquartile range of exam scores of AI and non-AI students are similar. This implies that, in the range where AI students and non-AI students have overlapping homework times, students who spend the same amount of time completing homework on average receive similar exam scores." "This pattern shows that students who spend the same amount of time on homework learn similarly, with or without generative AI. In other words, generative AI reduces time spent learning for the majority of AI students but not learning efficiency for those who spend the same time studying as the non-AI students."

GPerson 1mo ago
Right I was not clear enough about separating my opinion from the findings.
doctorpangloss 1mo ago
well most university exams are designed to measure how much you study. so we didn't really need a study to tell us, "Exams continue to measure what they are designed to measure." they're not designed to measure general aptitude, or function as admissions criteria, or screen for job applications, or any other numerous things they are used for. there can be many questions of pedagogy. one of them is, what do our exams measure and how do we use them? professors who say, "My exam is designed to measure who studies, not be used for all these other purposes that they are actually used for" - I don't buy it. It's the same as late night comedians saying they are not responsible for solutions, even when spending 90% of their air time making political jokes. THIS is the pedagogical issue, that pedagogy has NEVER caught up with the scope of responsibilities. This is acute in STEM - I mean, the humanities departments are generally pretty well run, all things considered, in this regard. Generative AI is accelerating that pre-existing crisis.
- thaumasiotes 1mo ago
  > well most university exams are designed to measure how much you study. Huh? They're designed to measure how much you know. They can't see how much you study, nor would they have reason to be interested.
  - doctorpangloss 1mo ago
    at least in my experience in university - i didn't really ask this question, since it is obvious to me, but some students have asked it during lecture, or some instructors have volunteered the answer ahead of time - if you ask how to perform better on the exam, usually the instructors say, "here's what you should study." they never say, "know more." the thing i am talking about is consistent with the paper. really, your takeaway should be, exams can't see how much you know!
    - bonoboTP 1mo ago
      Because "know more" isn't actionable. Knowing more is achieved by studying but not necessarily more time spent studying, but well spent effort. Staring at the page for hours and saying "I don't understand" doesn't help. Solve exercise problems, explain the material to fellow students, discuss it with them, make mind maps, bullet point summaries, work through derivations step by step, etc. There are many techniques. At the end of the day though what matters is what you know. Furthermore, if it's a serious subject, it shouldn't matter whether you learned it from this teacher or from another school and teacher, as long as your knowledge is correct. Knowing the idiosyncracies of this particular teacher should not factor into the grade. A serious subject can be learned on one continent and examined on another. Bullshit courses are all about learning pet peeves and hobby horses of a particular teacher.
    - fl4regun 1mo ago
      You become a person who knows more, by studying more about the things you need to know.
    - bee_rider 1mo ago
      I would usually say something along the lines of “everything we covered is on the table” or “everything we covered since the last exam is on the table” depending on the nature of the test. That’s the same message as “know more” but I think it sounds politer.
bonoboTP 1mo ago
It would be good to see the effect of access to AI during preparation on those who previously achieved top 10% points in exams of similar topics before. I suspect they would benefit further.
- kzz102 1mo ago
  My apologies for coming off as over-enthusiastic, I am currently obsessed with this study. Here is another quote: "The negative learning effects are larger for students with higher initial achievement. The differences in the estimated full (6-10 month average) effects are substantial, with a 50% gap between the most negative effect (-24 percent) for the highest tercile and the least negative (-16 percent) for the lowest tercile. " Not top 10% as you asked, but the closest to what you asked. My working hypothesis is that top performance is highly correlated with willingness to work hard, and AI decreases the motivation to work hard.
  - bonoboTP 1mo ago
    Interesting. Top academic performance is mostly correlated with conscientiousness (willingness to work hard and keeping track of things) and intelligence. And I'd add motivation and interest to that too. I think if you take a physics class where the student is intelligent and intrinsically motivated through their own interest (I admit this is rare) then AI probably helps.
    - bcrosby95 1mo ago
      I think if they are intelligent, intrinsically motivated, and willing to pursue knowledge beyond what the class requires, it probably helps. That last part is key. Intrinsic motivation doesn't mean you pursue it outside normal bounds. My kid loves soccer, its her second favorite thing in the world, she has an absurdly high tolerance for physical discomfort while playing, but she doesn't play it at home. There's other things she's rather do, such as play with her toys. When you move the bar to something even less interesting to most kids like science, you're going to have a pretty huge falloff. You're basically selecting for kids who choose to do it in their spare time. I know a lot of smart kids (I run a boyscout troop, my wife a girlscout troop, both with lots of high achievers), and none of them do this.
jonahx 1mo ago
Depends who's using it. Like many tools the force multiplier depends on the operator. I'm confident it's an amplifier for people who know how learning works and already do a lot of it, successfully. However the level of "learning fluency" I'm talking about isn't reached for many until late college or grad school, and sometimes not at all. So I'm not surprised by the quoted results for 12-18 year olds.
- bawolff 1mo ago
  And people who want to learn. Most teenagers lack agency in their studies. They aren't in high school because they love it but because they have no choice.
- ChrisSD 1mo ago
  Sometimes things are just common sense pure and simple.
  - dan_ggggg 1mo ago
    > Sometimes things are just common sense pure and simple. The discussion is about "AI", so common sense is out the window. These people's professional reputations depend on addict-level "AI" usage remaining socially acceptable.
- sebastiennight 1mo ago
  > Depends who's using it. Like many tools the force multiplier depends on the operator. How would you prove/disprove this assumption without falling into a True Scotsman fallacy?
  - jonahx 1mo ago
    Fair question. In theory you could do an experiment where the subjects were grad students or professors, or top performing college students. Give them some fixed amount of time to understand some new subject on which they'll be tested, only 1 group has access to an LLM with appropriate context for the learning task, etc. That's just one idea...
asdfman123 1mo ago
Personally in my own work it's pretty obvious how it's an amplifier. You waste far less time trying to find an answer to something that confuses you. You might say "it's good to learn research skills" and that's true to an extent, but tutors have always made people better students. And AI is a tutor you can message at any time, day or night, for free.
- nullsanity 1mo ago
  [dead]
- Barrin92 1mo ago
  the difference in the vast majority of cases is that people go to a tutor with an intentional stance of cultivating a particular kind of practice, not merely to get an answer to a question. It's not technically impossible to do this with a chatbot but far less likely given that, unlike an even half decent tutor, a chatbot will never cultivate that attitude in you. The elimination of that friction is exactly why people talk to bots. The good thing about a tutor is that he or she isn't always available and knows they won't be in the future, so they instill good habits and independence in you.
  - embedding-shape 1mo ago
    > a chatbot will never cultivate that attitude in you Hence the whole "AI is basically an amplifier of bad and good" argument made earlier. The ones who do this because they want to understand, don't need to cultivate this at all, it naturally happens with chatbots. They don't just ask for the answer to a question, but then dig into why it's like that and what not. But the ones that don't care, now have to do even less to get the fast answer without understanding.
- ungreased0675 1mo ago
  Yes, but is it the correct answer? I wasted several days trying to have AI teach me containers. I would have been much better off just reading the docs.
stopIiiiit 1mo ago
Self reports of AI use is worthless Social stigma against just juices people to stick to socially desirable answers; no doubt a whole bunch who self reported as non-AI users actually used AI
- myhf 1mo ago
  Sounds like the opposite of the social stigma in tech companies, where most people are under pressure to self-report as AI users when they are actually just using normal methods to get the same result (and spending the time saved however they like) (nothing against genAI for making work take more time, it's still good for business as long as the extra work can be moved around)
nilamo 1mo ago
I'm curious what happened in the source, that a copy/paste exposed such odd artifacts. "e§ort" == effort "Öve" == not sure. Maybe "over five"?
- zamfi 1mo ago
  I wondered the same. ff and fi are common ligature letter pairs, maybe the original is in an encoding where those are represented with separate characters, which then correspond to § and Ö in whatever encoding we find here?
  - messe 1mo ago
    They look more like OCR errors than encoding errors.
- kzz102 1mo ago
  It was copied from a pdf, maybe LaTeX generated. All the ligatures are wrong -- not sure where the problem lies, the pdf itself, pdf reader, or the clipboard on Linux.
- yencabulator 1mo ago
  PDF is a render-only format with optional hidden metadata for semantics. The PDF file they created knows how to paint the ligatures they used, like "fi" and "ff", but doesn't contain the `/ToUnicode` metadata for what those symbols meant. The generated PDF happened to put the "fi" ligature at an index that is normally used "Ö", and didn't provide more information.

by kzz102 1mo ago

GPerson 1mo ago

Right I was not clear enough about separating my opinion from the findings.

doctorpangloss 1mo ago

well most university exams are designed to measure how much you study. so we didn't really need a study to tell us, "Exams continue to measure what they are designed to measure." they're not designed to measure general aptitude, or function as admissions criteria, or screen for job applications, or any other numerous things they are used for. there can be many questions of pedagogy. one of them is, what do our exams measure and how do we use them? professors who say, "My exam is designed to measure who studies, not be used for all these other purposes that they are actually used for" - I don't buy it. It's the same as late night comedians saying they are not responsible for solutions, even when spending 90% of their air time making political jokes. THIS is the pedagogical issue, that pedagogy has NEVER caught up with the scope of responsibilities. This is acute in STEM - I mean, the humanities departments are generally pretty well run, all things considered, in this regard. Generative AI is accelerating that pre-existing crisis.

thaumasiotes 1mo ago
> well most university exams are designed to measure how much you study. Huh? They're designed to measure how much you know. They can't see how much you study, nor would they have reason to be interested.
- doctorpangloss 1mo ago
  at least in my experience in university - i didn't really ask this question, since it is obvious to me, but some students have asked it during lecture, or some instructors have volunteered the answer ahead of time - if you ask how to perform better on the exam, usually the instructors say, "here's what you should study." they never say, "know more." the thing i am talking about is consistent with the paper. really, your takeaway should be, exams can't see how much you know!
  - bonoboTP 1mo ago
    Because "know more" isn't actionable. Knowing more is achieved by studying but not necessarily more time spent studying, but well spent effort. Staring at the page for hours and saying "I don't understand" doesn't help. Solve exercise problems, explain the material to fellow students, discuss it with them, make mind maps, bullet point summaries, work through derivations step by step, etc. There are many techniques. At the end of the day though what matters is what you know. Furthermore, if it's a serious subject, it shouldn't matter whether you learned it from this teacher or from another school and teacher, as long as your knowledge is correct. Knowing the idiosyncracies of this particular teacher should not factor into the grade. A serious subject can be learned on one continent and examined on another. Bullshit courses are all about learning pet peeves and hobby horses of a particular teacher.
  - fl4regun 1mo ago
    You become a person who knows more, by studying more about the things you need to know.
  - bee_rider 1mo ago
    I would usually say something along the lines of “everything we covered is on the table” or “everything we covered since the last exam is on the table” depending on the nature of the test. That’s the same message as “know more” but I think it sounds politer.

bonoboTP 1mo ago

It would be good to see the effect of access to AI during preparation on those who previously achieved top 10% points in exams of similar topics before. I suspect they would benefit further.

kzz102 1mo ago
My apologies for coming off as over-enthusiastic, I am currently obsessed with this study. Here is another quote: "The negative learning effects are larger for students with higher initial achievement. The differences in the estimated full (6-10 month average) effects are substantial, with a 50% gap between the most negative effect (-24 percent) for the highest tercile and the least negative (-16 percent) for the lowest tercile. " Not top 10% as you asked, but the closest to what you asked. My working hypothesis is that top performance is highly correlated with willingness to work hard, and AI decreases the motivation to work hard.
- bonoboTP 1mo ago
  Interesting. Top academic performance is mostly correlated with conscientiousness (willingness to work hard and keeping track of things) and intelligence. And I'd add motivation and interest to that too. I think if you take a physics class where the student is intelligent and intrinsically motivated through their own interest (I admit this is rare) then AI probably helps.
  - bcrosby95 1mo ago
    I think if they are intelligent, intrinsically motivated, and willing to pursue knowledge beyond what the class requires, it probably helps. That last part is key. Intrinsic motivation doesn't mean you pursue it outside normal bounds. My kid loves soccer, its her second favorite thing in the world, she has an absurdly high tolerance for physical discomfort while playing, but she doesn't play it at home. There's other things she's rather do, such as play with her toys. When you move the bar to something even less interesting to most kids like science, you're going to have a pretty huge falloff. You're basically selecting for kids who choose to do it in their spare time. I know a lot of smart kids (I run a boyscout troop, my wife a girlscout troop, both with lots of high achievers), and none of them do this.

jonahx 1mo ago

Depends who's using it. Like many tools the force multiplier depends on the operator. I'm confident it's an amplifier for people who know how learning works and already do a lot of it, successfully. However the level of "learning fluency" I'm talking about isn't reached for many until late college or grad school, and sometimes not at all. So I'm not surprised by the quoted results for 12-18 year olds.

bawolff 1mo ago
And people who want to learn. Most teenagers lack agency in their studies. They aren't in high school because they love it but because they have no choice.
ChrisSD 1mo ago
Sometimes things are just common sense pure and simple.
- dan_ggggg 1mo ago
  > Sometimes things are just common sense pure and simple. The discussion is about "AI", so common sense is out the window. These people's professional reputations depend on addict-level "AI" usage remaining socially acceptable.
sebastiennight 1mo ago
> Depends who's using it. Like many tools the force multiplier depends on the operator. How would you prove/disprove this assumption without falling into a True Scotsman fallacy?
- jonahx 1mo ago
  Fair question. In theory you could do an experiment where the subjects were grad students or professors, or top performing college students. Give them some fixed amount of time to understand some new subject on which they'll be tested, only 1 group has access to an LLM with appropriate context for the learning task, etc. That's just one idea...

asdfman123 1mo ago

Personally in my own work it's pretty obvious how it's an amplifier. You waste far less time trying to find an answer to something that confuses you. You might say "it's good to learn research skills" and that's true to an extent, but tutors have always made people better students. And AI is a tutor you can message at any time, day or night, for free.

nullsanity 1mo ago
[dead]
Barrin92 1mo ago
the difference in the vast majority of cases is that people go to a tutor with an intentional stance of cultivating a particular kind of practice, not merely to get an answer to a question. It's not technically impossible to do this with a chatbot but far less likely given that, unlike an even half decent tutor, a chatbot will never cultivate that attitude in you. The elimination of that friction is exactly why people talk to bots. The good thing about a tutor is that he or she isn't always available and knows they won't be in the future, so they instill good habits and independence in you.
- embedding-shape 1mo ago
  > a chatbot will never cultivate that attitude in you Hence the whole "AI is basically an amplifier of bad and good" argument made earlier. The ones who do this because they want to understand, don't need to cultivate this at all, it naturally happens with chatbots. They don't just ask for the answer to a question, but then dig into why it's like that and what not. But the ones that don't care, now have to do even less to get the fast answer without understanding.
ungreased0675 1mo ago
Yes, but is it the correct answer? I wasted several days trying to have AI teach me containers. I would have been much better off just reading the docs.

stopIiiiit 1mo ago

Self reports of AI use is worthless Social stigma against just juices people to stick to socially desirable answers; no doubt a whole bunch who self reported as non-AI users actually used AI

myhf 1mo ago
Sounds like the opposite of the social stigma in tech companies, where most people are under pressure to self-report as AI users when they are actually just using normal methods to get the same result (and spending the time saved however they like) (nothing against genAI for making work take more time, it's still good for business as long as the extra work can be moved around)

nilamo 1mo ago

I'm curious what happened in the source, that a copy/paste exposed such odd artifacts. "e§ort" == effort "Öve" == not sure. Maybe "over five"?

zamfi 1mo ago
I wondered the same. ff and fi are common ligature letter pairs, maybe the original is in an encoding where those are represented with separate characters, which then correspond to § and Ö in whatever encoding we find here?
- messe 1mo ago
  They look more like OCR errors than encoding errors.
kzz102 1mo ago
It was copied from a pdf, maybe LaTeX generated. All the ligatures are wrong -- not sure where the problem lies, the pdf itself, pdf reader, or the clipboard on Linux.
yencabulator 1mo ago
PDF is a render-only format with optional hidden metadata for semantics. The PDF file they created knows how to paint the ligatures they used, like "fi" and "ff", but doesn't contain the `/ToUnicode` metadata for what those symbols meant. The generated PDF happened to put the "fi" ligature at an index that is normally used "Ö", and didn't provide more information.