7 ms·

Kids who use ChatGPT as a study assistant do worse on tests

176 points by notamy 2y ago

jdeaton 2y ago
Why is the study specifically of Turkish students
- eesmith 2y ago
  The paper at https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4895486 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4895486 says: > One of the co-authors, Özge Kabakcı, a high school math teacher and former department chair of the math department at our partner Turkish high school, led the development of all session materials.
- elashri 2y ago
  I guess it would be hard to find a high school with the sample size that you need (thousand) that will agree on collaborating. And in the US every county will have different rules and in terms of math they don't teach it in standard way. But why Turkish not British or any other place is going to be a question no matter the location. But do you really think the results will be significantly different if it is done lets say on Vietnamese students?
wiradikusuma 2y ago
Kids who have their parents do their homework do worse on tests. s/parents/chatgpt
- ggm 2y ago
  "I blame the parents" s/parents/chatgpt/
- vsuperpower2021 2y ago
  [flagged]
  - topato 2y ago
    Did you read it? When not expressly forbidden from doing so with the 'fine-tuned on their specific test problems and answers' model, they simply asked ChatGPT for the answer.
- eesmith 2y ago
  Do kids who have get private tutoring also average worse on tests? Your characterization of "do their homework" seems wrong as the paper says of the GPT Tutor branch that "is hard for students to use it as a crutch since its prompt asks it to avoid giving them the answer and instead guide them in a step-by-step fashion."
sim7c00 2y ago
why is this surprising. all such tools hamper learning. if you want to learn, read books, read and write. don't use a spellchecker for ur language exam. no calculator for calculus. pen and paper. how is this going backwards :(
- dools 2y ago
  I can report that I have learned more about bash scripting in the past couple of years using ChatGPT to write all manner of scripts, than I did in the previous 20 years of copying and pasting things off StackOverflow.
  - JTyQZSnP3cQGa8B 2y ago
    You’re supposed to learn by reading books and doing exercises by yourself (https://tldp.org/LDP/abs/html/ https://tldp.org/LDP/abs/html/). SO is equivalent to ChatGPT and will teach you nothing deep.
    - dools 2y ago
      Disagree that it's the equivalent of SO. Unlike SO, I can ask it why it is doing something and as a result I can better understand what it writes, I can create little test cases, and make sure it's not going to do anything destructive or incorrect. Given the fact that I only ever write bash scripts when I need to "make computer do thing now now now now now" the likelihood that I would ever take the time to deeply learn the ins and outs of bash are slim to none. I can ask it to explain certain constructs and give me worked examples very quickly to improve my understanding. Even though I can't say I could reproduce all the various things that it has written for me, I know things that bash can do that I was never aware of before, and I can recognise lots of things in the syntax now that would previously have looked like gibberish.
    - HKH2 2y ago
      Yeah, we all learn exactly the same way. We're all great at reading books and following steps in the right order.
    - 2y ago
lll-o-lll 2y ago
When I was young, and learning math, my father always forbade me from looking at the answer in the back of the textbook. “You don’t work backwards from the answer!”, and I think this is right. In life, we rarely have the answer in front of us, we have to work that out from the things we know. It’s this struggling that builds a muscle you can then apply to any problem. ChatGPT, I suspect, is akin to looking up the answer. You’re failing to exercise the muscle needed to solve novel (to you), problems.
- ShadowBlades512 2y ago
  I think this is very correct for studying, when I was in undergrad, if I saw the answer, I found it was more effective to skip past it and solve it later. Even more important, once I am done, I should not go look at the answer to confirm if I am right or wrong, I should try and validate the answer by looking at my solution and trying to figure out if my solution is correct or not because there are plenty of ways to "disqualify" an answer, once I learned to do this well, my grades really went up. However, I don't always agree with the fact that we don't have the answer in front of us in many situations. There are a lot of situations beyond grade school and undergrad where you do have the answer in front of you. Sometimes it's in for form of Numpy or Matlab or Simulink. It might be someone's research publication. Replicating these by working both forwards and backwards from their libraries or results can be much more effective.
  - LeonB 2y ago
    Good point about disproving your answer (before seeing the alleged real answer). The ability to quickly verify/disprove the answer is a separate and useful skill all by itself, that isn’t explicitly taught. Troubleshooting/problem solving, working backwards from symptom to problem etc — that’s another, richer and very rewarding “real world” type of skill that isn’t directly taught, (though often encountered incidentally).
  - wongarsu 2y ago
    Half of modern mathematics is basically assuming something is true and then trying to work forwards and backwards to show that it is true. But that requires a lot of rigor. If you hand-wave a single step you are just exercising an advanced version of confirmation bias. Working forwards and sanity checking your result is a lot more forgiving.
ggm 2y ago
I was willing to entertain the idea they could do better. I guess the tests have to be written to leverage the skill. That said, all things being equal kids who write notes by hand out-perform kids who type them. Even touch type them. So maybe the old ways are better in this specific brain-knowledge-competency-understanding forming space?
- cmdpint 2y ago
  I never was good at hand written notes, but I always outperformed my classmates on tests. Except for the one geometry class where the teacher gave me detention for not taking enough notes. Maybe I'm an exception.
  - ggm 2y ago
    I'm unsure its safe to generalize in this space TBH. I think your story is familiar and enough seen it might be a counter-case to the argument. That said.. "learning styles" has mostly been debunked. In truth we don't entirely know why some people do amazingly well in maths and some don't. It's an interesting field of research (and the related field of "does well in compsci")
thatkid02 2y ago
its like gulping food; one has to chew. Time to learn how to educate when knowledge is at your fingertips.
deleted 2y ago
[deleted]
gitroom 2y ago
What were the primary reasons that made students who used ChatGPT do poorly on math assessments, even though they had worked correctly through a greater number of practice problems?
- al_borland 2y ago
  They didn’t have to figure out how to solve the problem. Instead of struggling a bit, which is where the learning happens, they would likely go to ChatGPT for the answer. When the answer bot was taken away, they weren’t prepared to think about how to solve the problem and work it out. I’ve noticed this even using Copilot in VS Code. I rarely use it, but if I start pulling it out, I notice at the first hint of actually thinking about how to do something, my brain seeks to ask Copilot instead. It’s like there is an off switch that gets flicked when there is an easy button available. If I were to figure it out on my own, I’d know what to do next time I run into a problem like that… if I use Copilot, I’ve learned nothing, other than next time I run into this, use Copilot. It’s a crutch when it comes to learning.
  - sweeter 2y ago
    Exactly. I stopped using it entirely, and I noticed that I would write something and then pause, expecting copilot to take over for a while after quitting. It felt like my brain wasn't really engaged. Its anecdotal, but I feel a lot better after giving it up and I think I can do a lot more when I can fully reason about the problem after poking at it from different angles.
- JTyQZSnP3cQGa8B 2y ago
  > they had worked correctly ChatGPT worked correctly for them but they learned nothing. It should be pretty obvious that you don’t learn by copying answers.
darthrupert 2y ago
Did nobody read the article? It says right there that the students who used chatgpt right, as a tutor, did much better than their peers. If your human tutors just give you the answers when you ask for them, how do you think it'll ho?
- cybersquare 2y ago
  [flagged]
- metaphor 2y ago
  > It says right there that the students who used chatgpt right, as a tutor, did much better than their peers. No LLM processing required, just old fashioned critical reading and comprehension (my emphasis added): >> The students who used it did spectacularly better on the practice problems, solving 127 percent more of them correctly compared with students who did their practice work without any high-tech aids. But on a test afterwards, these AI-tutored students did no better.
alberth 2y ago
I have wondered if future generations will struggling with critical thinking / problem solving - without the aided technology assistance.
- klodolph 2y ago
  Current and previous generations struggle too. Critical thinking and problem solving skills have always been an absolute bear to teach, and few teachers or school systems have the stomach to teach it. I’m thinking of science education, for example. It’s not just a matter of teaching kids a bunch of disconnected facts about science like “the mitochondria is the powerhouse of the cell” but teaching kids to use the scientific method. To teach the scientific method well, you have to get the kids to design experiments and be comfortable with the prospect of failure. This is, sadly, a very modern approach and requires training the teachers how to do this. Some schools don’t but it’s not something you would have likely gotten much of in the 20th century. Instead, people are still fighting over dumb shit like which facts are taught in class. “Critical thinking skills” are not on the menu most of the time.
- al_borland 2y ago
  I think they will be the case. When I was a kid my high school math class was very calculator driven. I have a lot to say about that, but won’t bore you with it. I would often see adults pull out a pencil and paper and do a whole bunch of math very quickly, or do a lot of math quickly in their head. I couldn’t do this as well as them, and my math classes weren’t doing anything to move me in that direction. When the calculators were taken away, most of the kids were useless. I saw a guy in National Math Honor Society put 1-1 in his calculator, unironically. This was almost 25 years ago. I assume it’s only gotten worse, and will continue to get worse as more tools that provide answers are given to the students. I think new tools are great, but it seems to me that the people who can utilize them the best are the ones who learned without them. Those are the people who understand the concepts behind them, so they can ask the right questions, with the right terminology, and they can recognize if an answer makes sense. When learning is done with these tools, even just calculators, little is being learned. Memorize the button sequence to press leading up to the test. Once the test is over, forget the sequence… and even if the buttons are relented, it won’t matter, because while we all walk around with a calculator in our pockets now, it’s not a TI-83 (or whatever people use now), so without the conceptual knowledge, what good is the button sequence. That doesn’t even prepare a person to leverage ChatGPT well.
DeepYogurt 2y ago
Gasp
ignoramous 2y ago
The title could be worded better. Kids using "base" GPT4 performed poorly but the ones with access to a finely-tuned "tutor" GPT4 did okay. The study was purposefully done in a domain the current SoTA LLMs struggle in (Math). From the (draft!) paper's abstract: A key remaining question is how generative AI affects learning, namely, how humans acquire new skills as they perform tasks. This kind of skill learning is critical to long-term productivity gains, especially in domains where generative AI is fallible and human experts must check its outputs. .. Consistent with prior work, our results show that access to GPT-4 significantly improves performance (48% improvement for GPT Base and 127% for GPT Tutor). However, we additionally find that when access is subsequently taken away, students actually perform worse than those who never had access (17% reduction for GPT Base). That is, access to GPT-4 can harm educational outcomes. These negative learning effects are largely mitigated by the safeguards included in GPT Tutor. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4895486 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4895486
- tripletao 2y ago
  To emphasize, the "GPT Tutor" kids didn't do worse on the exam than the control kids, but they didn't do better either. The effect was slightly negative, statistically insignificant: > Student performance in the GPT Tutor cohort was statistically indistinguishable from that of the control cohort, and the point estimate was smaller by an order of magnitude (-0.004), suggesting minimal impact to performance in the unassisted exam. "GPT Base" would provide complete answers, but "GPT Tutor" was prompted to provide only hints. So the result is perhaps that: 1. Given the option to let a machine ("GPT Base") do their homework, many kids will lazily take it. These kids won't learn as much. 2. A machine that refuses to do their homework ("GPT Tutor") doesn't cause that problem. It doesn't seem to help either, though. I'd guess that laziness would explain most of the harm rather than mistakes made by "GPT Base", though I have no particular evidence for that. Maybe someone will repeat this study in a domain where "GPT Base" makes fewer mistakes, allowing those two effects to be distinguished. (Though would that pass ethics review, now that "GPT Base" is known to impair learning in at least some cases?)
hilbert42 2y ago
I learned the hard way: no pain, no gain.
deleted 2y ago
[deleted]
madhatter999 2y ago
From the abstract: “Consistent with prior work, our results show that access to GPT-4 significantly improves performance (48% improvement for GPT Base and 127% for GPT Tutor). However, we additionally find that when access is subsequently taken away, students actually perform worse than those who never had access (17% reduction for GPT Base).” Kids who use ChatGPT do actually “significantly” better according to the authors. Now I don’t know if significantly means statistically significant here because I haven’t read the methodology but 127% increase in performance must be something. That said, that’s a clickbaity title if I’ve ever seen one. Edit: Upon closer reading, the increase in performance is statistically significant. Also “access to GPT“ in this case is having GPT open while solving the problems, not studying with GPT and then solving the problems, which was my first understanding from the clickbaity title. Results are not terribly surprising in that regard.
- elashri 2y ago
  You can read the paper itself to get my useful information. The title and article is confusing. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4895486 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4895486
- alabhyajindal 2y ago
  They don't do better as far as learning is concerned. Isn't this concerning? > However, we additionally find that when access is subsequently taken away, students actually perform worse than those who never had access (17% reduction for GPT Base). The abstract continues: > That is, access to GPT-4 can harm educational outcomes. These negative learning effects are largely mitigated by the safeguards included in GPT Tutor. Our results suggest that students attempt to use GPT-4 as a "crutch" during practice problem sessions, and when successful, perform worse on their own. Thus, to maintain long-term productivity, we must be cautious when deploying generative AI to ensure humans continue to learn critical skills.
  - madhatter999 2y ago
    Relying on GPT while solving the problems must be inflating the grades (my use of the word inflate is intentional, because the evaluation does not represent the true knowledge of the student), which then results in lower retention in the long run.
alabhyajindal 2y ago
I recently used AI assistants for help with programming homework. My usual prompts include "help me think in the right direction", "is my thinking correct" etc. I also find myself copy pasting a question in chat to understand it better. I had the suspicion that this is not aiding in my learning process even though I am able to "solve" more problems. Nice to see this confirmed. Time to stop!
- dailykoder 2y ago
  I learned programming before genAI, but with plenty of video tutorials and stackoverflow available. Someday I noticed that I would automatically hit up the search engine within seconds when I hit a road block. It's kinda the same as with GenAI. You just have to know how to query search engines efficiently. I got stuff done, but I often did not learn much and sometimes didn't even understand what I was doing. So I started to force myself to use data- or reference sheets and try to come up with a solution myself. And if that wasn't good enough or I had real trouble, then I tried using some different resource. And it improved my programming experience tremendously. It's hard, but it's worth it.
  - tomcam 2y ago
    I am super intuitive and also suspect that you probably coded frequently. In fact, I would assumed you did so on… a daily basis ;)
- onehair 2y ago
  > I also find myself copy pasting a question in chat to understand it better. This is different fromt students relying on ChatGPT to pass tests. Your use is much more adulty and you're trying to understand before proceeding. You use ChatGPT like a tutor rather than a calculator, which is improving what you know rather than taking from it
  - alabhyajindal 2y ago
    Thanks! I think the problem is when I ask for assistance. It's perfectly fine to reach for help after a fair attempt. However, I sometimes catch myself reaching for help too quickly. This happens mostly when I'm tired which leads me to think of homework as something to get done with, rather than a learning exercise.
makk 2y ago
What if the test is irrelevant to the current times? “Those with ChatGPT solved 48 percent more of the practice problems correctly, but they ultimately scored 17 percent worse on a test of the topic that the students were learning.” So, in the real world, where people can use chatgpt in their jobs, the kids that use it will do better than the kids who don’t. Maybe a better test is: can you catch chatgpt when it is wrong? Not, can you answer without ChatGPT?
- goatlover 2y ago
  ChatGPT won't be able to help with all situations in the real world though. And there's a chance OpenAI fails and other LLMs become less accessible and more expensive. Future lawsuits and legislation could lead to them becoming cripled. It's a hell of a crutch to teach the next generation to lean on.
- gotoeleven 2y ago
  [flagged]
- jxyxfinite 2y ago
  I’d argue the opposite, that rarely in the real world is chatGPT easily accessible/usable. Working with a customer/client? I don’t think you’ll take a minute to say “give me one second let me ask ChatGPT”
  - eks391 2y ago
    Depends on the job. My sister uses an AI (Gemini) so much that she has a subscription. Her husband also uses it frequently, and keeps it open in a tab at all times. He is a programmer, and she does marketing. I don't use an LLM for work, but very easily could if I wanted. All three of us mostly communicate with coworkers/partners/clients through email, so it's definitely doable.
golergka 2y ago
> A third group of students had access to a revised version of ChatGPT that functioned more like a tutor. This chatbot was programmed to provide hints without directly divulging the answer. The students who used it did spectacularly better on the practice problems, solving 127 percent more of them correctly compared with students who did their practice work without any high-tech aids. Is it me, or is does this directly contradicts the title?
- al_borland 2y ago
  You seemed to leave the part out of your quote where it says those higher percentage of practice problems didn’t translate to test scores. At best they do the same, at the worst they do worse. There was no scenario where ChatGPT actually helped (unless you count practice problems, which are apparently a meaningless stat, since it doesn’t translate to test scores). Most kids in the general public would use normal ChatGPT, not one specially optimized to act like a tutor. So in the real world, kids who use it will do worse, and it’s not with seeking out the special one, as there is no advantage. The title seems like a pretty fair 1 line summary, and pours some much needed cold water on a space that has been full of hype and promises, many of which have fallen flat.
- noisy_boy 2y ago
  That 127 is when they had access to chatGPT during _practice_. They fared worse once that access was taken away during the actual exam.
ern 2y ago
I use ChatGPT 4o to check my child's homework, but I forbid them from using it directly. That way, I can make sure the work is correct (or at least wrong in the same way as ChatGPT) without straining my tired brain.
tj444 2y ago
The evaluation method is wrong. It's like when cars first came out, you ask people to drive cars for a month and they get used to cars. Then you ask them to compete in a horse race and see how fast they can go. We should evaluate how fast they solve a problem, no matter how.
fzeindl 2y ago
Side note and blog promotion: I find fascinated that ChatGPT can easily simulate the age of child when giving answers for homework: https://www.fabianzeindl.com/posts/chatgpt-simulating-agegroups https://www.fabianzeindl.com/posts/chatgpt-simulating-agegro...
dghlsakjg 2y ago
Used incorrectly. Yes. LLMs, for me, have been tremendously useful in learning new concepts. I frequently feed it my own notes and ask it to correct any misunderstandings, or to expand on things I don’t understand. I use it like I would an on demand tutor, but I can totally understand how it could be used as a shortcut that wouldn’t be helpful. In the same way, I can hire a tutor that will help me actually learn, or I can hire a “tutor” that just does the homework for me. I’ve worked as a tutor so I’ve seen people looking for both, and people that don’t want to learn are always going to find a way. People who do want to learn are also going to find a way.
- JumpCrisscross 2y ago
  > frequently feed it my own notes and ask it to correct any misunderstandings, or to expand on things I don’t understand In what fields? I’ve tried this with some simple finance and aerospace problems; it’s me, sophomore year undergrad, except someone laced everything I drank with LSD.
  - arrowsmith 2y ago
    I find it's good at answering my questions about humanities subjects e.g. history. Maybe it's only as good as an undergraduate, but I don't have a humanities degree, so an undergraduate is someone I can learn from.
    - JumpCrisscross 2y ago
      If it’s leading you to sources, go for it. If you’re taking it point blank, consider what you’re learning. (If it isn’t pointing you to sources, ask for them.)
  - blitzar 2y ago
    > it’s me sophomore year undergrad, laced with LSD That was the deepest most insightful version of me.
  - dghlsakjg 2y ago
    Technical fields. Right now I'm going through the Dive Into Deep Learning course/textbook. Oftentimes it will gloss over a concept, or assume knowledge. For a specific example: I wanted an expansion of why there are different loss functions and why you might choose one over another since the source material sort of plowed right through it. I got a good answer to my question without having to spend an hour reading through other materials
sirspacey 2y ago
[flagged]
- opdahl 2y ago
  Well I flagged your comment so right back at you.
eks391 2y ago
If I lived before the tape measure was invented, and rely on carefully placing my metersticks to measure things, I can get really good at measuring without the need for a measuring tape. After all, a measuring tape is just a few flexible metersticks anyways, so if you need to measure something longer than the full length of the tape, you are screwed. If you take the measuring tape away from the person who relied on that tool instead of being good at using a meterstick, or perhaps no tools besides their own arm length, they are gonna suddenly not be able to measure, unless they go through the effort of learning to measure without the tape. You can argue that measuring tape is a crutch preventing people from learning how to properly measure, and has its own limitations, but regardless its still really helpful, especially for people who only need to measure things occassionally, and not super long things. ChatGPT is a tool. Just like all other tools, like computers, cars, etc., if you take it away, most people cannot perform the function for which they relied on the tool to help them do.
- cageface 2y ago
  What if the function they were using the tool for is basic reasoning?
  - eks391 2y ago
    ChatGPT is not the first tool used to replace basic reasoning, and we haven't collapsed yet. I would argue that nearly all tools can be used to remove basic reasoning, depending on if you choose to use it in such a manner.
    - cageface 2y ago
      What previous tools would you say replaced basic reasoning? I would not classify calculators, spreadsheets, spelling and grammar checkers etc as basic reasoning tools. I'm talking about bringing together several independent concepts or sets of facts and forming a coherent point of view or argument from them.
      - eks391 2y ago
langsoul-com 2y ago
Not surprising. Test is all about memorising things. If you don't need to memorise everything because it's on google, you won't. Thus, when the test rolls around, nothing is memorised and then they do bad. It's like memorising phone numbers VS keeping in the contacts app. Before I memorised tons of numbers, but now they're all on the app and I barely recall my own.
valval 2y ago
I wouldn’t be terribly concerned about that, as testing as it’s done in school is a moronic practice to begin with.
hi_hi 2y ago
Story time. I always struggled with math as a kid. School to high school, then didn't touch it much until Uni. Teachers typically couldn't explain things in a way I "got it" in a school setting. I had some success with a private tutor to get me over the line in high school. Then at Uni I'm doing Computer Graphics, which included advanced (for me) math. I was panicked, and initially struggled until one of my good friends who was also studying the same course, and is VERY good at math, was able to answer my vague "I don't get it" questions, or at least guide me to more specific questions. I think I'm quite a visual learner, I don't think at that time there was a concept of people learning "differently". Luckily my good friend was also a visual learner, along with also being very good at math. It was like someone was able to see how my brain worked and feed me information in a way it could compile. I became quite good at math after that. You really need to learn how to learn. Its fascinating, but also horrifying when I now consider all the lives that have been negatively impacted because this wasn't understood, and people were led to believe they couldn't do something which maybe then really wanted to be able to do. If GenAI can help with that, I'm all in.
- OccamsMirror 2y ago
  How did your friend teach you math visually?
  - hi_hi 2y ago
    I'm using the term "visual" very loosely here. I prefer to be able to conceptually group and link things logically. He was able to explain things in a context that matched this, and greatly simplified things. Some things that I still suck at learning are foreign languages. Sometimes there are rules you can follow, sometimes you just have to "know", and sometimes the rules that apply in one case are the opposite in another. I'm amazed anyone can communicate effectively at all :-)
    - Sammi 2y ago
      Sounds like your friend is an amazing tutor. Communication and teaching is indeed hard. I really believe most people don't learn math because they don't get the teaching they need, not because math is somehow cognitively inaccessible for them (except in case of low iq). Lots of people learn language and grammar and are able to write strong texts with a high level of rational and logical thinking, but have not been able to learn math. These people clearly have the mental faculties for math, they just didn't get the teaching they needed. Math is a language for expressing thought, just like any language is, and these people have mastered other languages, so they have the prerequisites for being able to learn the language of math. Math is an especially hard language and it requires good teaching.
Jiahang 2y ago
learning need focus
tripletao 2y ago
These comments are filled with misunderstandings of the result. There were three groups of kids: 1. Control, with no LLM assistance at any time. 2. "GPT Base", raw ChatGPT as provided by OpenAI. 3. "GPT Tutor", improved by the researchers to provide hints rather than complete answers and to make fewer mistakes on their specific problems. On study problem sets ("as a study assistant"), kids with access to either GPT did better than control. When GPT access was subsequently removed from all participants ("on tests"), the kids who studied with "GPT Base" did worse than control. The kids with "GPT Tutor" were statistically indistinguishable from control.
- lyu07282 2y ago
  Reading comprehension is really awful nowadays or people tend to just comprehend what confirms their prior beliefs. The sad part is that none of those people will ever realize the errors in their comprehension of the article. That's exactly one of the mechanisms how people form wrong opinions, it's compounding and almost impossible to change. The takeaway should be (besides the research still needing reproduction) to encourage the control of the type of AI agent that is given to students, ones that don't just give answers to copy but provide tutoring. OpenAI should be forced to develop such a "student mode" immediately and parents and educators need to be made aware of it to make sure students are using it, otherwise students are going to get much worse in tests, as they just ask it for answers to copy in assignments.
- Jensson 2y ago
  Changing things almost always improves results, that is the first rule you need to remember during education testing. Most of the improvements disappear when you make it standard. This effect likely comes from novelty being more interesting so kids gets more alert, but when they are used to it then it is the same old boring thing and education results go back to normal. Of course things can improve or get worse, but in general it is really hard to say, you need to have a massive advantage over the standard during testing to actually get any real improvements, most of the time you just make things worse.
- blitzar 2y ago
  Assuming that the kids with "Human Tutor" were statistically better than control (they were not in the study so we will not know) - this is a very poor showing for ChatGPT.
brianhama 2y ago
But better in life…
godelski 2y ago
Are people not reading the article here? Let me tldr: - Study had 3 groups: normal GPT, system prompt to make GPT act as tutor and focus on giving hints, not answers, and no GPT Group 1 (normal GPT) - 48% better on practice problems - 17% worse on test Group 2 (tutor GPT) - 127% better on practice problems - equal test score to control group GPT errors: - 50% error rate - 8% error on arithmetic problems - step by step instructions we're wrong 42% of time - GPT tutor was fed answers - students with GPT and GPT tutor predicted that they did better (so both groups were over confident) Paper: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4895486 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4895486 I'll reply with my opinion to this comment. But many comments are not responding to the article content
- godelski 2y ago
  I think the part of overconfidence is actually important here. Looking at the paper Figure 2 shows the number of questions. For all sessions with GPT students averaged 2 messages per problem. Interestingly 20% isn't restating the problem or asking for the answer. Looks like 50-60% of tutor questions are asking for answer or restating problem and they ask more questions. In my experience, usually where GPT and other LLMs get things wrong is in the steps. I see people are frequently over confident about the results and I've long thought this was a big part of it that the answer may be right but steps to get there are wrong. I see this a lot with river crossing problems. People will often show a prompt that solves it (half the time there's information leakage) and a good portion of those will have the logic errors. But people just look at the answer (btw, this is the reason I claim LLMs don't reason. It's not about getting the answer wrong, it's about the logical steps. That's the real evaluation!) Personally I think this is good evidence of over fitting (memorization). I also think this is why you should be careful when using it to code. It's the details that matter a lot. Since LLMs are aggregators I'd like to remind everyone that the average coder and average code is terrible. Hell, even good programmers often suck. Code is hard! Whenever I try using LLMs to code I find that I write lines faster but I end up spending more time debugging and prompt engineering than if I just read the docs and did it myself (this also has the added benefit of the struggle making me remember more). So I'll use it to write things I don't really care about but otherwise I'm not getting how people are finding it so helpful (yes, I read my comment). But I suspect many people are similar to the students in the study.
dorkwood 2y ago
[dead]
jgb1984 2y ago
I have a visceral dislike, even hate, for what the LLM hype brought the world. The never ending slop it is spouting, filling up the entire internet. More and more I get confronted with images and media that turn out to be AI generated, when I find out I am disgusted and just close the tab. Soulless drivel, endlessly streaming. And I'm confident that the education system as we know it will be severely damaged because of it. Even in our own field, I can guarantee you that software developers that "grew up" with these garbage AI assistants will be worse coders than the generation that came before. You will never develop the understanding, the insight, that's needed by chatgpt'ing your way through college and life. Excellent news for my own market value of course, but I don't hesitate to say that I regret the LLM hype happened, the impact on the world is overwhelmingly negative (not even touching on the catastrophic environmental and financial cost to society).
namaria 2y ago
I liken using llms for 'studying' to going to a gym with a hydraulic lift. Yeah you'll lift much, much more. But is that the point?
seeg 2y ago
I think the problem that people don't see anymore is using tests themselves. A clever idea is worth more than a single tick in the correct checkbox. This applies to maths as well. Tests are faster to check and, supposedly, objective, but a viva voce exam is still superior imho.
JSDevOps 2y ago
Yeah because it lies and makes stuff up to fill in any gaps.
RachealMuldoon 2y ago
[dead]
BlueTemplar 2y ago
> A draft paper about the experiment was posted on the website of SSRN, formerly known as the Social Science Research Network, in July 2024. The paper has not yet been published in a peer-reviewed journal and could still be revised. Should have started with that. A study without independent replication hardly counts as «researchers found», much less one that hadn't even been peer-reviewed yet !
JeremyNT 2y ago
It seems kind of obvious, no? The act of repetition and processing the data ourselves is what leads to a deeper understanding, and asking a chatbot for an answer seems like it would skip the thinking required when learning "the old fashioned way." Maybe we can learn how to incorporate using chatbots in education, but I suspect there need to be guardrails on when and how they are used so students can get the benefit of doing the work themselves.
tmaly 2y ago
I think the way to us ChatGPT is to have it explain a concept once and give a few examples. After that, the student should struggle the old fashion way with problems. I would like to see a study that looks at this approach.
plusfour 2y ago
Test results are a measure of how well you can do on tests.

176 points by notamy 2y ago

jdeaton 2y ago

Why is the study specifically of Turkish students

eesmith 2y ago
The paper at https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4895486 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4895486 says: > One of the co-authors, Özge Kabakcı, a high school math teacher and former department chair of the math department at our partner Turkish high school, led the development of all session materials.
elashri 2y ago
I guess it would be hard to find a high school with the sample size that you need (thousand) that will agree on collaborating. And in the US every county will have different rules and in terms of math they don't teach it in standard way. But why Turkish not British or any other place is going to be a question no matter the location. But do you really think the results will be significantly different if it is done lets say on Vietnamese students?

wiradikusuma 2y ago

Kids who have their parents do their homework do worse on tests. s/parents/chatgpt

ggm 2y ago
"I blame the parents" s/parents/chatgpt/
vsuperpower2021 2y ago
[flagged]
- topato 2y ago
  Did you read it? When not expressly forbidden from doing so with the 'fine-tuned on their specific test problems and answers' model, they simply asked ChatGPT for the answer.
eesmith 2y ago
Do kids who have get private tutoring also average worse on tests? Your characterization of "do their homework" seems wrong as the paper says of the GPT Tutor branch that "is hard for students to use it as a crutch since its prompt asks it to avoid giving them the answer and instead guide them in a step-by-step fashion."

sim7c00 2y ago

why is this surprising. all such tools hamper learning. if you want to learn, read books, read and write. don't use a spellchecker for ur language exam. no calculator for calculus. pen and paper. how is this going backwards :(

dools 2y ago
I can report that I have learned more about bash scripting in the past couple of years using ChatGPT to write all manner of scripts, than I did in the previous 20 years of copying and pasting things off StackOverflow.
- JTyQZSnP3cQGa8B 2y ago
  You’re supposed to learn by reading books and doing exercises by yourself (https://tldp.org/LDP/abs/html/ https://tldp.org/LDP/abs/html/). SO is equivalent to ChatGPT and will teach you nothing deep.
  - dools 2y ago
    Disagree that it's the equivalent of SO. Unlike SO, I can ask it why it is doing something and as a result I can better understand what it writes, I can create little test cases, and make sure it's not going to do anything destructive or incorrect. Given the fact that I only ever write bash scripts when I need to "make computer do thing now now now now now" the likelihood that I would ever take the time to deeply learn the ins and outs of bash are slim to none. I can ask it to explain certain constructs and give me worked examples very quickly to improve my understanding. Even though I can't say I could reproduce all the various things that it has written for me, I know things that bash can do that I was never aware of before, and I can recognise lots of things in the syntax now that would previously have looked like gibberish.
  - HKH2 2y ago
    Yeah, we all learn exactly the same way. We're all great at reading books and following steps in the right order.
  - 2y ago

lll-o-lll 2y ago

When I was young, and learning math, my father always forbade me from looking at the answer in the back of the textbook. “You don’t work backwards from the answer!”, and I think this is right. In life, we rarely have the answer in front of us, we have to work that out from the things we know. It’s this struggling that builds a muscle you can then apply to any problem. ChatGPT, I suspect, is akin to looking up the answer. You’re failing to exercise the muscle needed to solve novel (to you), problems.

ShadowBlades512 2y ago
I think this is very correct for studying, when I was in undergrad, if I saw the answer, I found it was more effective to skip past it and solve it later. Even more important, once I am done, I should not go look at the answer to confirm if I am right or wrong, I should try and validate the answer by looking at my solution and trying to figure out if my solution is correct or not because there are plenty of ways to "disqualify" an answer, once I learned to do this well, my grades really went up. However, I don't always agree with the fact that we don't have the answer in front of us in many situations. There are a lot of situations beyond grade school and undergrad where you do have the answer in front of you. Sometimes it's in for form of Numpy or Matlab or Simulink. It might be someone's research publication. Replicating these by working both forwards and backwards from their libraries or results can be much more effective.
- LeonB 2y ago
  Good point about disproving your answer (before seeing the alleged real answer). The ability to quickly verify/disprove the answer is a separate and useful skill all by itself, that isn’t explicitly taught. Troubleshooting/problem solving, working backwards from symptom to problem etc — that’s another, richer and very rewarding “real world” type of skill that isn’t directly taught, (though often encountered incidentally).
- wongarsu 2y ago
  Half of modern mathematics is basically assuming something is true and then trying to work forwards and backwards to show that it is true. But that requires a lot of rigor. If you hand-wave a single step you are just exercising an advanced version of confirmation bias. Working forwards and sanity checking your result is a lot more forgiving.

ggm 2y ago

I was willing to entertain the idea they could do better. I guess the tests have to be written to leverage the skill. That said, all things being equal kids who write notes by hand out-perform kids who type them. Even touch type them. So maybe the old ways are better in this specific brain-knowledge-competency-understanding forming space?

cmdpint 2y ago
I never was good at hand written notes, but I always outperformed my classmates on tests. Except for the one geometry class where the teacher gave me detention for not taking enough notes. Maybe I'm an exception.
- ggm 2y ago
  I'm unsure its safe to generalize in this space TBH. I think your story is familiar and enough seen it might be a counter-case to the argument. That said.. "learning styles" has mostly been debunked. In truth we don't entirely know why some people do amazingly well in maths and some don't. It's an interesting field of research (and the related field of "does well in compsci")

thatkid02 2y ago

its like gulping food; one has to chew. Time to learn how to educate when knowledge is at your fingertips.

deleted 2y ago

[deleted]

gitroom 2y ago

What were the primary reasons that made students who used ChatGPT do poorly on math assessments, even though they had worked correctly through a greater number of practice problems?

al_borland 2y ago
They didn’t have to figure out how to solve the problem. Instead of struggling a bit, which is where the learning happens, they would likely go to ChatGPT for the answer. When the answer bot was taken away, they weren’t prepared to think about how to solve the problem and work it out. I’ve noticed this even using Copilot in VS Code. I rarely use it, but if I start pulling it out, I notice at the first hint of actually thinking about how to do something, my brain seeks to ask Copilot instead. It’s like there is an off switch that gets flicked when there is an easy button available. If I were to figure it out on my own, I’d know what to do next time I run into a problem like that… if I use Copilot, I’ve learned nothing, other than next time I run into this, use Copilot. It’s a crutch when it comes to learning.
- sweeter 2y ago
  Exactly. I stopped using it entirely, and I noticed that I would write something and then pause, expecting copilot to take over for a while after quitting. It felt like my brain wasn't really engaged. Its anecdotal, but I feel a lot better after giving it up and I think I can do a lot more when I can fully reason about the problem after poking at it from different angles.
JTyQZSnP3cQGa8B 2y ago
> they had worked correctly ChatGPT worked correctly for them but they learned nothing. It should be pretty obvious that you don’t learn by copying answers.

darthrupert 2y ago

Did nobody read the article? It says right there that the students who used chatgpt right, as a tutor, did much better than their peers. If your human tutors just give you the answers when you ask for them, how do you think it'll ho?

cybersquare 2y ago
[flagged]
metaphor 2y ago
> It says right there that the students who used chatgpt right, as a tutor, did much better than their peers. No LLM processing required, just old fashioned critical reading and comprehension (my emphasis added): >> The students who used it did spectacularly better on the practice problems, solving 127 percent more of them correctly compared with students who did their practice work without any high-tech aids. But on a test afterwards, these AI-tutored students did no better.

alberth 2y ago

I have wondered if future generations will struggling with critical thinking / problem solving - without the aided technology assistance.

klodolph 2y ago
Current and previous generations struggle too. Critical thinking and problem solving skills have always been an absolute bear to teach, and few teachers or school systems have the stomach to teach it. I’m thinking of science education, for example. It’s not just a matter of teaching kids a bunch of disconnected facts about science like “the mitochondria is the powerhouse of the cell” but teaching kids to use the scientific method. To teach the scientific method well, you have to get the kids to design experiments and be comfortable with the prospect of failure. This is, sadly, a very modern approach and requires training the teachers how to do this. Some schools don’t but it’s not something you would have likely gotten much of in the 20th century. Instead, people are still fighting over dumb shit like which facts are taught in class. “Critical thinking skills” are not on the menu most of the time.
al_borland 2y ago
I think they will be the case. When I was a kid my high school math class was very calculator driven. I have a lot to say about that, but won’t bore you with it. I would often see adults pull out a pencil and paper and do a whole bunch of math very quickly, or do a lot of math quickly in their head. I couldn’t do this as well as them, and my math classes weren’t doing anything to move me in that direction. When the calculators were taken away, most of the kids were useless. I saw a guy in National Math Honor Society put 1-1 in his calculator, unironically. This was almost 25 years ago. I assume it’s only gotten worse, and will continue to get worse as more tools that provide answers are given to the students. I think new tools are great, but it seems to me that the people who can utilize them the best are the ones who learned without them. Those are the people who understand the concepts behind them, so they can ask the right questions, with the right terminology, and they can recognize if an answer makes sense. When learning is done with these tools, even just calculators, little is being learned. Memorize the button sequence to press leading up to the test. Once the test is over, forget the sequence… and even if the buttons are relented, it won’t matter, because while we all walk around with a calculator in our pockets now, it’s not a TI-83 (or whatever people use now), so without the conceptual knowledge, what good is the button sequence. That doesn’t even prepare a person to leverage ChatGPT well.

DeepYogurt 2y ago

Gasp

ignoramous 2y ago

The title could be worded better. Kids using "base" GPT4 performed poorly but the ones with access to a finely-tuned "tutor" GPT4 did okay. The study was purposefully done in a domain the current SoTA LLMs struggle in (Math). From the (draft!) paper's abstract: A key remaining question is how generative AI affects learning, namely, how humans acquire new skills as they perform tasks. This kind of skill learning is critical to long-term productivity gains, especially in domains where generative AI is fallible and human experts must check its outputs. .. Consistent with prior work, our results show that access to GPT-4 significantly improves performance (48% improvement for GPT Base and 127% for GPT Tutor). However, we additionally find that when access is subsequently taken away, students actually perform worse than those who never had access (17% reduction for GPT Base). That is, access to GPT-4 can harm educational outcomes. These negative learning effects are largely mitigated by the safeguards included in GPT Tutor. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4895486 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4895486

tripletao 2y ago
To emphasize, the "GPT Tutor" kids didn't do worse on the exam than the control kids, but they didn't do better either. The effect was slightly negative, statistically insignificant: > Student performance in the GPT Tutor cohort was statistically indistinguishable from that of the control cohort, and the point estimate was smaller by an order of magnitude (-0.004), suggesting minimal impact to performance in the unassisted exam. "GPT Base" would provide complete answers, but "GPT Tutor" was prompted to provide only hints. So the result is perhaps that: 1. Given the option to let a machine ("GPT Base") do their homework, many kids will lazily take it. These kids won't learn as much. 2. A machine that refuses to do their homework ("GPT Tutor") doesn't cause that problem. It doesn't seem to help either, though. I'd guess that laziness would explain most of the harm rather than mistakes made by "GPT Base", though I have no particular evidence for that. Maybe someone will repeat this study in a domain where "GPT Base" makes fewer mistakes, allowing those two effects to be distinguished. (Though would that pass ethics review, now that "GPT Base" is known to impair learning in at least some cases?)

hilbert42 2y ago

I learned the hard way: no pain, no gain.

deleted 2y ago

[deleted]

madhatter999 2y ago

From the abstract: “Consistent with prior work, our results show that access to GPT-4 significantly improves performance (48% improvement for GPT Base and 127% for GPT Tutor). However, we additionally find that when access is subsequently taken away, students actually perform worse than those who never had access (17% reduction for GPT Base).” Kids who use ChatGPT do actually “significantly” better according to the authors. Now I don’t know if significantly means statistically significant here because I haven’t read the methodology but 127% increase in performance must be something. That said, that’s a clickbaity title if I’ve ever seen one. Edit: Upon closer reading, the increase in performance is statistically significant. Also “access to GPT“ in this case is having GPT open while solving the problems, not studying with GPT and then solving the problems, which was my first understanding from the clickbaity title. Results are not terribly surprising in that regard.

elashri 2y ago
You can read the paper itself to get my useful information. The title and article is confusing. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4895486 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4895486
alabhyajindal 2y ago
They don't do better as far as learning is concerned. Isn't this concerning? > However, we additionally find that when access is subsequently taken away, students actually perform worse than those who never had access (17% reduction for GPT Base). The abstract continues: > That is, access to GPT-4 can harm educational outcomes. These negative learning effects are largely mitigated by the safeguards included in GPT Tutor. Our results suggest that students attempt to use GPT-4 as a "crutch" during practice problem sessions, and when successful, perform worse on their own. Thus, to maintain long-term productivity, we must be cautious when deploying generative AI to ensure humans continue to learn critical skills.
- madhatter999 2y ago
  Relying on GPT while solving the problems must be inflating the grades (my use of the word inflate is intentional, because the evaluation does not represent the true knowledge of the student), which then results in lower retention in the long run.

alabhyajindal 2y ago

I recently used AI assistants for help with programming homework. My usual prompts include "help me think in the right direction", "is my thinking correct" etc. I also find myself copy pasting a question in chat to understand it better. I had the suspicion that this is not aiding in my learning process even though I am able to "solve" more problems. Nice to see this confirmed. Time to stop!

dailykoder 2y ago
I learned programming before genAI, but with plenty of video tutorials and stackoverflow available. Someday I noticed that I would automatically hit up the search engine within seconds when I hit a road block. It's kinda the same as with GenAI. You just have to know how to query search engines efficiently. I got stuff done, but I often did not learn much and sometimes didn't even understand what I was doing. So I started to force myself to use data- or reference sheets and try to come up with a solution myself. And if that wasn't good enough or I had real trouble, then I tried using some different resource. And it improved my programming experience tremendously. It's hard, but it's worth it.
- tomcam 2y ago
  I am super intuitive and also suspect that you probably coded frequently. In fact, I would assumed you did so on… a daily basis ;)
onehair 2y ago
> I also find myself copy pasting a question in chat to understand it better. This is different fromt students relying on ChatGPT to pass tests. Your use is much more adulty and you're trying to understand before proceeding. You use ChatGPT like a tutor rather than a calculator, which is improving what you know rather than taking from it
- alabhyajindal 2y ago
  Thanks! I think the problem is when I ask for assistance. It's perfectly fine to reach for help after a fair attempt. However, I sometimes catch myself reaching for help too quickly. This happens mostly when I'm tired which leads me to think of homework as something to get done with, rather than a learning exercise.

makk 2y ago

What if the test is irrelevant to the current times? “Those with ChatGPT solved 48 percent more of the practice problems correctly, but they ultimately scored 17 percent worse on a test of the topic that the students were learning.” So, in the real world, where people can use chatgpt in their jobs, the kids that use it will do better than the kids who don’t. Maybe a better test is: can you catch chatgpt when it is wrong? Not, can you answer without ChatGPT?

goatlover 2y ago
ChatGPT won't be able to help with all situations in the real world though. And there's a chance OpenAI fails and other LLMs become less accessible and more expensive. Future lawsuits and legislation could lead to them becoming cripled. It's a hell of a crutch to teach the next generation to lean on.
gotoeleven 2y ago
[flagged]
jxyxfinite 2y ago
I’d argue the opposite, that rarely in the real world is chatGPT easily accessible/usable. Working with a customer/client? I don’t think you’ll take a minute to say “give me one second let me ask ChatGPT”
- eks391 2y ago
  Depends on the job. My sister uses an AI (Gemini) so much that she has a subscription. Her husband also uses it frequently, and keeps it open in a tab at all times. He is a programmer, and she does marketing. I don't use an LLM for work, but very easily could if I wanted. All three of us mostly communicate with coworkers/partners/clients through email, so it's definitely doable.

golergka 2y ago

> A third group of students had access to a revised version of ChatGPT that functioned more like a tutor. This chatbot was programmed to provide hints without directly divulging the answer. The students who used it did spectacularly better on the practice problems, solving 127 percent more of them correctly compared with students who did their practice work without any high-tech aids. Is it me, or is does this directly contradicts the title?

al_borland 2y ago
You seemed to leave the part out of your quote where it says those higher percentage of practice problems didn’t translate to test scores. At best they do the same, at the worst they do worse. There was no scenario where ChatGPT actually helped (unless you count practice problems, which are apparently a meaningless stat, since it doesn’t translate to test scores). Most kids in the general public would use normal ChatGPT, not one specially optimized to act like a tutor. So in the real world, kids who use it will do worse, and it’s not with seeking out the special one, as there is no advantage. The title seems like a pretty fair 1 line summary, and pours some much needed cold water on a space that has been full of hype and promises, many of which have fallen flat.
noisy_boy 2y ago
That 127 is when they had access to chatGPT during _practice_. They fared worse once that access was taken away during the actual exam.

ern 2y ago

I use ChatGPT 4o to check my child's homework, but I forbid them from using it directly. That way, I can make sure the work is correct (or at least wrong in the same way as ChatGPT) without straining my tired brain.

tj444 2y ago

The evaluation method is wrong. It's like when cars first came out, you ask people to drive cars for a month and they get used to cars. Then you ask them to compete in a horse race and see how fast they can go. We should evaluate how fast they solve a problem, no matter how.

fzeindl 2y ago

Side note and blog promotion: I find fascinated that ChatGPT can easily simulate the age of child when giving answers for homework: https://www.fabianzeindl.com/posts/chatgpt-simulating-agegroups https://www.fabianzeindl.com/posts/chatgpt-simulating-agegro...

dghlsakjg 2y ago

Used incorrectly. Yes. LLMs, for me, have been tremendously useful in learning new concepts. I frequently feed it my own notes and ask it to correct any misunderstandings, or to expand on things I don’t understand. I use it like I would an on demand tutor, but I can totally understand how it could be used as a shortcut that wouldn’t be helpful. In the same way, I can hire a tutor that will help me actually learn, or I can hire a “tutor” that just does the homework for me. I’ve worked as a tutor so I’ve seen people looking for both, and people that don’t want to learn are always going to find a way. People who do want to learn are also going to find a way.

JumpCrisscross 2y ago
> frequently feed it my own notes and ask it to correct any misunderstandings, or to expand on things I don’t understand In what fields? I’ve tried this with some simple finance and aerospace problems; it’s me, sophomore year undergrad, except someone laced everything I drank with LSD.
- arrowsmith 2y ago
  I find it's good at answering my questions about humanities subjects e.g. history. Maybe it's only as good as an undergraduate, but I don't have a humanities degree, so an undergraduate is someone I can learn from.
  - JumpCrisscross 2y ago
    If it’s leading you to sources, go for it. If you’re taking it point blank, consider what you’re learning. (If it isn’t pointing you to sources, ask for them.)
- blitzar 2y ago
  > it’s me sophomore year undergrad, laced with LSD That was the deepest most insightful version of me.
- dghlsakjg 2y ago
  Technical fields. Right now I'm going through the Dive Into Deep Learning course/textbook. Oftentimes it will gloss over a concept, or assume knowledge. For a specific example: I wanted an expansion of why there are different loss functions and why you might choose one over another since the source material sort of plowed right through it. I got a good answer to my question without having to spend an hour reading through other materials

sirspacey 2y ago

[flagged]

opdahl 2y ago
Well I flagged your comment so right back at you.

eks391 2y ago

If I lived before the tape measure was invented, and rely on carefully placing my metersticks to measure things, I can get really good at measuring without the need for a measuring tape. After all, a measuring tape is just a few flexible metersticks anyways, so if you need to measure something longer than the full length of the tape, you are screwed. If you take the measuring tape away from the person who relied on that tool instead of being good at using a meterstick, or perhaps no tools besides their own arm length, they are gonna suddenly not be able to measure, unless they go through the effort of learning to measure without the tape. You can argue that measuring tape is a crutch preventing people from learning how to properly measure, and has its own limitations, but regardless its still really helpful, especially for people who only need to measure things occassionally, and not super long things. ChatGPT is a tool. Just like all other tools, like computers, cars, etc., if you take it away, most people cannot perform the function for which they relied on the tool to help them do.

cageface 2y ago
What if the function they were using the tool for is basic reasoning?
- eks391 2y ago
  ChatGPT is not the first tool used to replace basic reasoning, and we haven't collapsed yet. I would argue that nearly all tools can be used to remove basic reasoning, depending on if you choose to use it in such a manner.
  - cageface 2y ago
    What previous tools would you say replaced basic reasoning? I would not classify calculators, spreadsheets, spelling and grammar checkers etc as basic reasoning tools. I'm talking about bringing together several independent concepts or sets of facts and forming a coherent point of view or argument from them.
    - eks391 2y ago

langsoul-com 2y ago

Not surprising. Test is all about memorising things. If you don't need to memorise everything because it's on google, you won't. Thus, when the test rolls around, nothing is memorised and then they do bad. It's like memorising phone numbers VS keeping in the contacts app. Before I memorised tons of numbers, but now they're all on the app and I barely recall my own.

valval 2y ago

I wouldn’t be terribly concerned about that, as testing as it’s done in school is a moronic practice to begin with.

hi_hi 2y ago

Story time. I always struggled with math as a kid. School to high school, then didn't touch it much until Uni. Teachers typically couldn't explain things in a way I "got it" in a school setting. I had some success with a private tutor to get me over the line in high school. Then at Uni I'm doing Computer Graphics, which included advanced (for me) math. I was panicked, and initially struggled until one of my good friends who was also studying the same course, and is VERY good at math, was able to answer my vague "I don't get it" questions, or at least guide me to more specific questions. I think I'm quite a visual learner, I don't think at that time there was a concept of people learning "differently". Luckily my good friend was also a visual learner, along with also being very good at math. It was like someone was able to see how my brain worked and feed me information in a way it could compile. I became quite good at math after that. You really need to learn how to learn. Its fascinating, but also horrifying when I now consider all the lives that have been negatively impacted because this wasn't understood, and people were led to believe they couldn't do something which maybe then really wanted to be able to do. If GenAI can help with that, I'm all in.

OccamsMirror 2y ago
How did your friend teach you math visually?
- hi_hi 2y ago
  I'm using the term "visual" very loosely here. I prefer to be able to conceptually group and link things logically. He was able to explain things in a context that matched this, and greatly simplified things. Some things that I still suck at learning are foreign languages. Sometimes there are rules you can follow, sometimes you just have to "know", and sometimes the rules that apply in one case are the opposite in another. I'm amazed anyone can communicate effectively at all :-)
  - Sammi 2y ago
    Sounds like your friend is an amazing tutor. Communication and teaching is indeed hard. I really believe most people don't learn math because they don't get the teaching they need, not because math is somehow cognitively inaccessible for them (except in case of low iq). Lots of people learn language and grammar and are able to write strong texts with a high level of rational and logical thinking, but have not been able to learn math. These people clearly have the mental faculties for math, they just didn't get the teaching they needed. Math is a language for expressing thought, just like any language is, and these people have mastered other languages, so they have the prerequisites for being able to learn the language of math. Math is an especially hard language and it requires good teaching.

Jiahang 2y ago

learning need focus

tripletao 2y ago

These comments are filled with misunderstandings of the result. There were three groups of kids: 1. Control, with no LLM assistance at any time. 2. "GPT Base", raw ChatGPT as provided by OpenAI. 3. "GPT Tutor", improved by the researchers to provide hints rather than complete answers and to make fewer mistakes on their specific problems. On study problem sets ("as a study assistant"), kids with access to either GPT did better than control. When GPT access was subsequently removed from all participants ("on tests"), the kids who studied with "GPT Base" did worse than control. The kids with "GPT Tutor" were statistically indistinguishable from control.

lyu07282 2y ago
Reading comprehension is really awful nowadays or people tend to just comprehend what confirms their prior beliefs. The sad part is that none of those people will ever realize the errors in their comprehension of the article. That's exactly one of the mechanisms how people form wrong opinions, it's compounding and almost impossible to change. The takeaway should be (besides the research still needing reproduction) to encourage the control of the type of AI agent that is given to students, ones that don't just give answers to copy but provide tutoring. OpenAI should be forced to develop such a "student mode" immediately and parents and educators need to be made aware of it to make sure students are using it, otherwise students are going to get much worse in tests, as they just ask it for answers to copy in assignments.
Jensson 2y ago
Changing things almost always improves results, that is the first rule you need to remember during education testing. Most of the improvements disappear when you make it standard. This effect likely comes from novelty being more interesting so kids gets more alert, but when they are used to it then it is the same old boring thing and education results go back to normal. Of course things can improve or get worse, but in general it is really hard to say, you need to have a massive advantage over the standard during testing to actually get any real improvements, most of the time you just make things worse.
blitzar 2y ago
Assuming that the kids with "Human Tutor" were statistically better than control (they were not in the study so we will not know) - this is a very poor showing for ChatGPT.

brianhama 2y ago

But better in life…

godelski 2y ago

Are people not reading the article here? Let me tldr: - Study had 3 groups: normal GPT, system prompt to make GPT act as tutor and focus on giving hints, not answers, and no GPT Group 1 (normal GPT) - 48% better on practice problems - 17% worse on test Group 2 (tutor GPT) - 127% better on practice problems - equal test score to control group GPT errors: - 50% error rate - 8% error on arithmetic problems - step by step instructions we're wrong 42% of time - GPT tutor was fed answers - students with GPT and GPT tutor predicted that they did better (so both groups were over confident) Paper: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4895486 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4895486 I'll reply with my opinion to this comment. But many comments are not responding to the article content

godelski 2y ago
I think the part of overconfidence is actually important here. Looking at the paper Figure 2 shows the number of questions. For all sessions with GPT students averaged 2 messages per problem. Interestingly 20% isn't restating the problem or asking for the answer. Looks like 50-60% of tutor questions are asking for answer or restating problem and they ask more questions. In my experience, usually where GPT and other LLMs get things wrong is in the steps. I see people are frequently over confident about the results and I've long thought this was a big part of it that the answer may be right but steps to get there are wrong. I see this a lot with river crossing problems. People will often show a prompt that solves it (half the time there's information leakage) and a good portion of those will have the logic errors. But people just look at the answer (btw, this is the reason I claim LLMs don't reason. It's not about getting the answer wrong, it's about the logical steps. That's the real evaluation!) Personally I think this is good evidence of over fitting (memorization). I also think this is why you should be careful when using it to code. It's the details that matter a lot. Since LLMs are aggregators I'd like to remind everyone that the average coder and average code is terrible. Hell, even good programmers often suck. Code is hard! Whenever I try using LLMs to code I find that I write lines faster but I end up spending more time debugging and prompt engineering than if I just read the docs and did it myself (this also has the added benefit of the struggle making me remember more). So I'll use it to write things I don't really care about but otherwise I'm not getting how people are finding it so helpful (yes, I read my comment). But I suspect many people are similar to the students in the study.

dorkwood 2y ago

[dead]

jgb1984 2y ago

I have a visceral dislike, even hate, for what the LLM hype brought the world. The never ending slop it is spouting, filling up the entire internet. More and more I get confronted with images and media that turn out to be AI generated, when I find out I am disgusted and just close the tab. Soulless drivel, endlessly streaming. And I'm confident that the education system as we know it will be severely damaged because of it. Even in our own field, I can guarantee you that software developers that "grew up" with these garbage AI assistants will be worse coders than the generation that came before. You will never develop the understanding, the insight, that's needed by chatgpt'ing your way through college and life. Excellent news for my own market value of course, but I don't hesitate to say that I regret the LLM hype happened, the impact on the world is overwhelmingly negative (not even touching on the catastrophic environmental and financial cost to society).

namaria 2y ago

I liken using llms for 'studying' to going to a gym with a hydraulic lift. Yeah you'll lift much, much more. But is that the point?

seeg 2y ago

I think the problem that people don't see anymore is using tests themselves. A clever idea is worth more than a single tick in the correct checkbox. This applies to maths as well. Tests are faster to check and, supposedly, objective, but a viva voce exam is still superior imho.

JSDevOps 2y ago

Yeah because it lies and makes stuff up to fill in any gaps.

RachealMuldoon 2y ago

[dead]

BlueTemplar 2y ago

> A draft paper about the experiment was posted on the website of SSRN, formerly known as the Social Science Research Network, in July 2024. The paper has not yet been published in a peer-reviewed journal and could still be revised. Should have started with that. A study without independent replication hardly counts as «researchers found», much less one that hadn't even been peer-reviewed yet !

JeremyNT 2y ago

It seems kind of obvious, no? The act of repetition and processing the data ourselves is what leads to a deeper understanding, and asking a chatbot for an answer seems like it would skip the thinking required when learning "the old fashioned way." Maybe we can learn how to incorporate using chatbots in education, but I suspect there need to be guardrails on when and how they are used so students can get the benefit of doing the work themselves.

tmaly 2y ago

I think the way to us ChatGPT is to have it explain a concept once and give a few examples. After that, the student should struggle the old fashion way with problems. I would like to see a study that looks at this approach.

plusfour 2y ago

Test results are a measure of how well you can do on tests.