16 ms·

> A common question is: “how much are students using AI to cheat?” That’s hard to answer, especially as we don’t know the specific educational context where eac

> A common question is: “how much are students using AI to cheat?” That’s hard to answer, especially as we don’t know the specific educational context where each of Claude’s responses is being used. I built a popular product that helps teachers with this problem. Yes, it's "hard to answer", but let's be honest... it's a very very widespread problem. I've talked to hundreds of teachers about this and it's a ubiquitous issue. For many students, it's literally "let me paste the assignment into ChatGPT and see what it spits out, change a few words and submit that". I think the issue is that it's so tempting to lean on AI. I remember long nights struggling to implement complex data structures in CS classes. I'd work on something for an hour before I'd have an epiphany and figure out what was wrong. But that struggling was ultimately necessary to really learn the concepts. With AI, I can simply copy/paste my code and say "hey, what's wrong with this code?" and it'll often spot it (nevermind the fact that I can just ask ChatGPT "create a b-tree in C" and it'll do it). That's amazing in a sense, but also hurts the learning process.

stv_123 2y ago
Yeah, the concept of "productive struggle" is important to the education process and having a way to short circuit it seems like it leads to worse learning outcomes.
- umpalumpaaa 2y ago
  I am not sure all humans work the same way though. Some get very very nervous when they begin to struggle. So nervous that they just stop functioning. I felt that during my time in university. I absolutely loved reading and working through dense math text books but the moment there was a time constraint the struggle turned into chaos.
  - AlexandrB 2y ago
    > Some get very very nervous when they begin to struggle. So nervous that they just stop functioning. I sympathize, but it's impossible to remove all struggle from life. It's better in the long run to work through this than try to avoid it.
yapyap 2y ago
> I think the issue is that it's so tempting to lean on AI. I remember long nights struggling to implement complex data structures in CS classes. I'd work on something for an hour before I'd have an epiphany and figure out what was wrong. But that struggling was ultimately necessary to really learn the concepts. With AI, I can simply copy/paste my code and say "hey, what's wrong with this code?" and it'll often spot it (nevermind the fact that I can just ask ChatGPT "create a b-tree in C" and it'll do it). That's amazing in a sense, but also hurts the learning process. In the end the willingness to struggle will set apart the truly great Software Engineer from the AI-crutched. Now of course this will most of the time not be rewarded, when a company looks at two people and sees “passable” code from both but one is way more “productive” with it (the AI-crutched engineer) they’ll inititally appreciate this one more. But in the long run they won’t be able to explain the choices made when creating the software, we will see the retraction from this type of coding when the first few companies’ security falls apart like a house of cards due to AI reliance. It’s basically the “instant gratification vs delayed gratification” argument but wrapped in the software dev box.
- JohnMakin 2y ago
  I don't wholly disagree with this post, but I'd like to add a caveat, observing my own workflow with these tools. I guess I'd qualify to you as someone "AI crutched" but I mostly use it for research and bouncing ideas (or code complete, which I've mentioned before - this is a great use of the tool and I wouldn't consider it a crutch, personally). For instance, "parse this massive log output, and highlight anything interesting you see or any areas that may be a problem, and give me your theories." Lots of times its wrong. Sometimes its right. Sometimes, its response gives me an idea that leads to another direction. It's essentially how I was using google + stack overflow ten years ago - see your list of answers, use your intuition, knowledge, and expertise to find the one most applicable to you, continue. This "crutch" is essentially the same one I've always used, just in different form. I find it pretty good at doing code review for myself before I submit something more formal, to catch any embarrassing or glaringly obvious bugs or incorrect test cases. I would be wary of the dev that refused to use tools out of some principled stand like this, just as I'd be wary of a dev that overly relied on them. There is a balance. Now, if all you know are these tools and the workflow you described, yea, that's probably detrimental to growth.
vunderba 2y ago
I've been calling this out since the rise of ChatGPT: "The real danger lies in their seductive nature - over how tempting it becomes to immediately reach for the LLM to provide an answer, rather than taking a few moments to quietly ponder the problem on your own. By reaching for it to solve any problem at nearly an instinctual level you are completely failing to cultivate an intrinsically valuable skill - that of critical reasoning."
- nonethewiser 2y ago
  Somewhat agree. I agree in principal - the process of problem solving is the important part. However I think LLMs make you do more of this because of what you can offload to the LLM. You can offload the simpler things. But for the complex questions that cut across multiple domains and have a lot of ambiguity? You're still going to have to sit down and think about it. Maybe once you've broken it into sufficiently smaller problems you can use the LLM. If we're worried about abstract problem solving skills that doesnt really go away with better tools. It goes away when we arent the ones using the tools.
  - Planktonne 2y ago
    You can offload the simpler things, but struggling with the simpler things is how you build the skills to handle the more complex ones that you can't hand off. If the simpler thing in question is a task you've already mastered, then you're not losing much by asking an LLM to help you with it. If it's not trivial to you though, then you're missing an opportunity to learn.
    - jplusequalt 2y ago
      Couldn't have said it better myself. The biology of the human brain will not change as a result of these LLMs. We are imperfect and will tend to take the easiest route in most cases. Having an "all powerful" tool that can offload the important work of figuring out tough problems seems like it will lead to a society less capable in solving complex problems.
    - 2y ago
bko 2y ago
When modern search became more available, a lot of people said there's no point of rote memorization as you can just do a Google search. That's more or less accepted today. Whenever we have a new technology there's a response "why do I need to learn X if I can always do Y", and more or less, it has proven true, although not immediately. For instance, I'm not too concerned about my child's ability to write very legibly (most writing is done on computers), spell very well (spell check keeps us professional), reading a map to get around (GPS), etc Not that these aren't noble things or worth doing, but they won't impact your life too much if you're not interest in penmanship, spelling, or cartography. I believe LLMs are different (I am still stuck in the moral panic phase), but I think my children will have a different perspective (similar to how I feel about memorizing poetry and languages without garbage collection). So how do I answer my child when he asks "Why should I learn to do X if I can just ask an LLM and it will do it better than me"
- nyeah 2y ago
  Spell check isn't really adequate. You get a page full of correctly spelled words, but they're the wrong words.
  - walthamstow 2y ago
    Try being British, often they're not correctly spelt words at all.
    - deleted 2y ago
      [deleted]
- delusional 2y ago
  "More or less" is doing a lot of work there. School, at least where I am, still spends the first year getting children to memorize the order of the numbers from 1-20 and if there's an even or odd number of a thing on a picture. Do you google if 5 is less than 6 or do you just memorize that? If you believe that creativity is not based on a foundation of memorization and experience (which is just memorization) you need to reflect on the connection between those.
taftster 2y ago
I don't think asking "what's wrong with my code" hurts the learning process. In fact, I would argue it helps it. I don't think you learn when you have reached your frustration point and you just want the dang assignment completed. But before reaching that point, if you had a tutor or assistant that you could ask, "hey, I'm just not seeing my mistake, do you have ideas" goes a long way to foster learning. ChatGPT, used in this way, can be extremely valuable and can definitely unlock learning in new ways which we probably even haven't seen yet. That being said, I agree with you, if you just ask ChatGPT to write a b-tree implementation from scratch, then you have not learned anything. So like all things in academia, AI can be used to foster education or cheat around it. There's been examples of these "cheats" far before ChatGPT or Google existed.
- SoftTalker 2y ago
  No I think the struggle is essential. If you can just ask a tutor (real or electronic) what is wrong with your code, you stop thinking and become dependent on that. Learning to think your way through a roadblock that seems like a showstopper is huge. It's sort of the mental analog of weight training. The only way to get better at weightlifting is to actually lift weight.
  - taftster 2y ago
    If I were to go and try to bench 300lbs, I would absolutely need a spotter to rescue me. Taking on more weight than I can possibly achieve is a setup for failure. Sure, I should probably practice benching 150lbs. That would be a good challenge for me and I would benefit from that experience. But 300lbs would crush me.
    - theamk 2y ago
      Sadly, ChatGPT is like a spotter that takes over at the smallest hint of struggle. Yes, you are not going to get crushed, but you won't get any workout done either. You really want start with a smaller weight, and increment it in steps as you progress. You know, like a class or something. And when you do those exercises, you really want to be lifting those weights yourself, and not rely on spotter for every rep.
Maskawanian 2y ago
Agreed, the only thing that is certain is that they are cheating themselves. While it can be useful to use LLMs as a tutor if you're stuck. The moment that you use it to provide a solution, you stop learning and the tool becomes a required stepping stone.
hobo_in_library 2y ago
The challenge is that while LLMs do not know everything, they are likely to know everything that's needed for your undergraduate education. So if you use them at that level you may learn the concepts at hand, but you won't learn _how to struggle_ to come up with novel answers. Then later in life when you actually hit problem domains that the LLM wasn't trained in, you'll not have learned the thinking patterns needed to persist and solve those problems. Is that necessarily a bad thing? It's mixed: - You lower the bar for entry for a certain class of roles, making labor cheaper and problems easier to solve at that level. - For more senior roles that are intrinsically solving problems without answers written in a book or a blog post somewhere, you need to be selective about how you evaluate the people who are ready to take on that role. It's like taking the college weed out classes and shifting those to people in the middle of their career. Individuals who can't make the cut will find themselves stagnating in their roles (but it'll also be easier for them to switch fields). Those who can meet the bar might struggle but can do well. Business will also have to come up with better ways to evaluate candidates. A resume that says "Graduated with a degree in X" will provide less of a signal than it did in the past
psygn89 2y ago
Agreed, the struggle often leads us to poke and prod an issue from many angles until things finally click. It lets us think critically. In that journey you might've learned other related concepts which further solidifies your understanding. But when the answer flows out of thin air right in front of you with AI, you get the "oh duh" or "that makes sense" moments and not the "a-ha" moment that ultimately sticks with you. Now does everything need an "a-ha" moment? No. However, I think core concepts and fundamentals need those "a-ha" moments to build a solid and in-depth foundation of understanding to build upon.
- taftster 2y ago
  Absolutely this. AI can help reveal solutions that weren't seen. An a-ha moment can be as instrumental to learning as the struggle that came before. Academia needs to embrace this concept and not try to fight it. AI is here, it's real, it's going to be used. Let's teach our students how to benefit from its (ethical) use.
- porridgeraisin 2y ago
  Yep. People love to cut down this argument by saying that a few decades ago, people said the same thing about calculators. But that was a problem too! People losing a large portion of their mental math faculty is definitely a problem. If mental math was required daily, we wouldn't see such obvious BS numbers in every kind of reporting(media/corporate/tech benchmarks) that people don't bat an eye at. How much the problem is _worth_ though, is what matters for adoption of these kinds of tech. Clearly, the problem above wasn't worth much. We now have to wait and see how much the "did not learn through cuts and scratches" problem is worth.
dyauspitr 2y ago
I’m pretty sure you can assume close to 100% of students are using LLMs to do their homework.
- ryandrake 2y ago
  And if you're that one person out of 100,000 who is not using LLMs to do their homework, you are at a significant disadvantage on the grading curve.
  - DontchaKnowit 2y ago
    Maybe, but piss on that, who needs good grades? Youll learn a hell of a lot better
  - Eavolution 2y ago
    Currently in university and my experience is that it heavily depends on the module. For a lot your statement's probably accurate, however for others it really isn't. For example we have a microprocessors module which is programming for an rp2040 in c, but also manually setting up interrupt handlers etc in assembly. All of the LLM's are completely useless for it, they tell you that the rp2040 works in ways it just doesn't and are actively unhelpful with the misinformation. The only students who can do well in that module are the ones that understand the material well and go to the datasheet and documentation instead of an LLM.
  - dysoco 2y ago
    My university solves this quite easily. There is no graded homework, the coursework is there only as a guide and practice for the exams. So you can absolutely use LLMs to help you with the exercises or to help understand something, however if you blindly get answers you will only be fooling yourself as you won't be able to pass the exams.
    - dyauspitr 2y ago
      That’s how most schooling has already been in a lot of South and East Asia. If you don’t do your homework, you get punished in other ways, but it doesn’t have any impact on the overall grade, the grade solely depends on the final exam.
andai 2y ago
I spent much of the past year at public libraries, and I heard the word ChatGPT approximately once per minute, in surround sound. Always from young people, and usually in a hushed tone...
moltar 2y ago
I think it’s finally time to just stop the homework. All school work must be done within the walls of the school. What are we teaching our children? It’s ok to do more work at home? There are countries that have no homework and they do just fine.
- jplusequalt 2y ago
  Homework helps reinforce the material learned in class. It's already a problem where there is too much material to be fit into a single class period. Trying to cram in enough time for homework will only make that problem worse.
  - moltar 2y ago
    Can do the work the next day to reinforce. As I said there are countries without homework and they seem to do ok. So it’s not mandatory by any means.
    - jplusequalt 2y ago
      >Can do the work the next day to reinforce. Keeping the curriculum fixed, there's already barely enough time to cover everything. Cutting the amount of lectures in half to make room for in-class homework time does not fix this fundamental problem.
      - DontchaKnowit 2y ago
        Just make lecture times longer.
        umbra07 2y ago
        students already don't pay attention in lecture: * due to either learning/concentration issues * the fact that most lecturers are boring, dull, and unengaging * and oftentimes you can learn better from other sources making lecture longer doesn't fix a single one of these issues. it just makes students learn even less.
srveale 2y ago
IMO it's so easy to ChatGPT your homework that the whole education model needs to flip on its head. Some teachers already do something like this, it's called the "Flipped classroom" approach. Basically, a student's marks depend mostly (only?) on what they can do in a setting where AI is verifiably unavailable. It means less class time for instruction, but students have a tutor in their pocket anyway. I've also talked with a bunch of teachers and a couple admins about this. They agree it's a huge problem. By the same token, they are using AI to create their lesson plans and assignments! Not fully of course, they edit the output using their expertise. But it's funny to imagine AI completing an AI assignment with the humans just along for the ride. The point is, if you actually want to know what a student is capable of, you need to watch them doing it. Assigning homework has lost all meaning.
- vonneumannstan 2y ago
  >Not fully of course, they edit the output using their expertise Surely this is sarcasm, but really your average schoolteacher is now a C student Education Major.
  - srveale 2y ago
    I was talking about people I know and talk with, mostly friends and family, who are smart, hard working, and their students are lucky to have them.
- hackyhacky 2y ago
  > it's called the "Flipped classroom" approach. Flipped classroom is just having the students give lectures, instead of the teacher. > Basically, a student's marks depend mostly (only?) on what they can do in a setting where AI is verifiably unavailable. This is called "proctored exams" and it's been pretty common in universities for a few centuries. None of this addresses the real issue, which is whether teachers should be preventing students from using AIs.
  - srveale 2y ago
    > Flipped classroom is just having the students give lectures, instead of the teacher. Not quite. Flipped classroom means more instruction outside of class time and less homework. > This is called "proctored exams" and it's been pretty common in universities for a few centuries. None of this addresses the real issue Proctored exams is part of it. In-class assignments is another. Asynchronous instruction is another. And yes, it addresses the issue. Students can use AI however they see fit, to learn or to accomplish tasks or whatever, but for actual assessment of ability they cannot use AI. And it leaves the door open for "open-book" exams where the use of AI is allowed, just like a calculator and textbook/cheat-sheet is allowed for some exams. https://en.wikipedia.org/wiki/Flipped_classroom https://en.wikipedia.org/wiki/Flipped_classroom
chalst 2y ago
Students who do that risk submitting assignments that show they don’t understand the course so far.
0xffff2 2y ago
>For many students, it's literally "let me paste the assignment into ChatGPT and see what it spits out, change a few words and submit that". Does that actually work? I'm long past having easy access to college programming assignments, but based on my limited interaction with ChatGPT I would be absolutely shocked if it produced output that was even coherent, much less working code given such an approach.
- StefanBatory 2y ago
  I have some subjects, at Masters - that are solvable by one prompt. One. Quality of CS/Software Engineering programs vary that much.
  - porridgeraisin 2y ago
    Yeah. On the other hand, "implement boruvkas MST algorithm in cuda such that only the while(numcomponents > 1) loop runs on the CPU, and everything else runs in the gpu. Memcpy everything onto the gpu first and only transfer back the count each iteration/keep it in pinned memory" It never gets it right, even after many reattempts in cursor. And even if it gets it right, it doesn't do the parallelization effectively enough - it's a hard problem to parallelize.
- bongodongobob 2y ago
  Why are you asking? Go try it. And yes, depending on the task, it does.
  - 0xffff2 2y ago
    As I said, I'm not a student, so I don't have access to a homework assignment to paste in. Ironically I have pretty much everything I ever submitted for my undergrad, but it seems like I absolutely never archived the assignments for some reason.
    - bongodongobob 2y ago
      I was able to get ~80% one shots on Advent of Code with 4o up to about day 12 iirc.
ryandrake 2y ago
I think teachers also need to reconsider how they are measuring mastery in the subject. LLMs exist. There is no putting the cat back into the bag. If your 1980s way to measure a student's mastery of a subject can be fooled by an LLM, then how effective is that measurement in 2020+? Maybe we need to stop using essays as a way to tell if the student has learned the material. Don't ask me what the solution is. Maybe your product does it. If I knew, I'd be making a fortune selling it to universities.
teekert 2y ago
Students do something akin to vibe coding I guess. It may seem impressive at first glance but if anything breaks you are so, so lost. Maybe that’s it, break the student’s code, see how they fix it. The vibe coding student is easily separate from the real one (of course this real coder can also use AI, just not yoloing it). I guess you can apply similar mechanics to reports. Some deeper questions and you will know if the report was self written or if an AI did it.
bboygravity 2y ago
I don't get this reasoning. Without LLMs I would learn how to write sub-optimal code that is somewhat functional. With LLMs instantly see "how it's done" for my exact problem case which makes me learn way faster. On top of that it always makes dumb mistakes which forces you to actually understand what it's spitting out to get it to work properly. Again: that helps with learning. The fact that you can ask it for a solution for exactly the context you're interested in is amazing and traditional learning doesn't come close in terms of efficiency IMO.
- dingnuts 2y ago
  > With LLMs instantly see "how it's done" for my exact problem case which makes me learn way faster. No, you see a plausible set of tokens that appear similar to how it's done, and as a beginner, you're not able to tell the difference between a good example and something that is subtly wrong. So you learn something, but it's wrong. You internalize it. Later, it comes back to bite you. But OpenAI keeps the money for the tokens. You pay whether the LLM is right or not. Sam likes that.
  - Spivak 2y ago
    This makes for a good sound bite but it's just not true. The use case of "show me what is a customary solution to <problem>" plays exactly into LLMs strength as a funny kind of search engine. I used to (and still do) search public code for this use case to get a sense of the style and idioms common in a new language/library and the plausible set of tokens is doing exactly that.
  - margalabargala 2y ago
    > So you learn something, [...] You internalize it. Or they don't.
- layer8 2y ago
  It’s more like looking up the solution to the math problem you’re supposed to solve on your own. It can be helpful in some situations, but in general you don’t learn the problem-solving skills if you don’t do it yourself.
deleted 2y ago
[deleted]
victorbjorklund 2y ago
In one way I'm glad I learned to code before LLM:s. It would be so hard to push through the learning now when you are just a click away from buildning the app with AI...
sally_glance 2y ago
I think this is a structural issue. Universities right now are trying to justify their existence - universities of the past used to be sites of innovation. Using ChatGPT doesn't dumb down your students. Not knowing how it works and where to use it does. Don't do silly textbook challenges for exams anymore - reestablish a culture of scientific innovation!
- acbart 2y ago
  Incorrect. Fundamentals must be taught in order to provide the context for the more challenging open-ended activities. Memorization is the base of knowledge, a starting point. Cheating (whether through an LLM or hiring someone or whatever) skips the journey. You can't just take them through the exciting routes, sometimes they have to go through the boring tedious repetitive stuff because that's how human brains learn. Learning is, literally, a stressful process on the brain. Students try to avoid it, but that's not good for them. At least in the introductory core classes.
  - sally_glance 2y ago
    I guess I should have phrased it differently - what I meant was just stop testing the tedious stuff, make it clear to students that learning the fundamentals is expected. Then examine them on hard exploratory problems which require the fundamentals.
- nixpulvis 2y ago
  You claim using AI tools doesn't dumb you down, but it very well could and is. Take the calculator for example, I'm overly dependent on it. I'm slower to perform arithmetic than I would have been without it. But knowing how to use one allows me to do more complex math more quickly. So I'm "dumber" in one way and "smarter" in others. AI could be the same... except our education system doesn't seem ready for it. We still learn arithmetic, even if we later rely in tools to do it. Right now teachers don't know how to teach so that AI doesn't trivialize things. You need to know how to do things so you know when the AI is lying to you.
  - 2y ago
aj7 2y ago
I’m a physicist. I can align and maximize ANY laser. I don’t even think when doing this task. Long hours of struggle, 50 years ago. Without struggle there is nothing. You can bullshit your way in. But you will be ejected.
- ketzo 2y ago
  barely related to your point but “I can align and maximize ANY laser” is such an incredibly specific flex, I love it
  - ramraj07 2y ago
    Especially because it's not a skill everyone gets just because they practice. I know because I tried for years lol.
- marksbrown 2y ago
  A master blacksmith can shoe a horse an' all. Laser alignment is also a solved problem with a machine. Just because something can be done by hand does not mean it has any intrinsic value.
whatever1 2y ago
The counter argument is that now you can skip boilerplate code and focus on the overall design and the few points that brainpower is really needed. The amount of visualizations that i have made after chat gpt was released has increased exponentially. I loath looking the documentation again and again to make a slightly non standard graph. Now all of the friction is gone! Graphs and visuals are everywhere in my code!
- hansvm 2y ago
  > focus on [...] the few points that brainpower is really needed The person you're responding to is talking about it from an educational perspective though. If your fundamentals aren't solid, you won't know that exponentially smoothed reservoir sampling backed by a splay tree is optimal for your problem, and ChatGPT has no clue either. Trying things, struggling, and failing is crucial to efficient learning. Not to mention, you need enough brain power or expertise to know when it's bullshitting you. Just today it was telling me that a packed array was better than my proposed solution, confidently explaining why, and not once saying anything correct. No prompt changes could fix it (whether restarting or replying), and anyone who tried to use less brainpower there would be up a creek when their solution sucked. Mind you, I use LLMs a lot, including for code-adjacent tasks and occasionally for code itself. It's a neat tool. It has its place though, and it must be used correctly.
musicale 2y ago
> “how much are students using AI to cheat?” That’s hard to answer "It is difficult to get a man to understand something, when his salary depends on his not understanding it!"
enjo 2y ago
> it's literally "let me paste the assignment into ChatGPT and see what it spits out, change a few words and submit that". My wife is an accounting professor. For many years her battle was with students using Chegg and the like. They would submit roughly correct answers but because she would rotate the underlying numbers they would always be wrong in a provably cheating way. This made up 5-8% of her students. Now she receives a parade of absolutely insane answers to questions from a much larger proportion of her students (she is working on some research around this but it's definitely more than 30%). When she asks students to recreate how they got to these pretty wild answers they never have any ability to articulate what happened. They are simply throwing her questions at LLMs and submitting the output. It's not great.
- DSingularity 2y ago
  This is now reality -- fighting to change the students is a losing battle. Besides in terms of normalizing grade distributions this is not that complicated to solve. Target the cheaters with pop quizzes. Prof can randomly choose 3 questions from assignments. If students cant get enough marks on 2/3 of them they are dealt a huge penalty. Students that actually work through the problems will have no problems with scoring enough marks on 2/3 of the questions. Students that lean irresponsibly on LLMs will lose their marks.
  - rrr_oh_man 2y ago
    Maybe we'll revert to Soviet bilet-style oral exams...
    - skandinaff 2y ago
      But they do that already in Danish universities, and I have to admit, those are very effective.
  - cellularmitosis 2y ago
    Why not just grade solely based on live performance? (quizzes and tests) Homework would still be assigned as a learning tool, but has no impact on your grade.
ugh123 2y ago
>I built a popular product that helps teachers with this problem. Does your product help teachers detect cheating? Because I hear none of them are accurate, with many false positives and ruined academic careers. Are you saying yours is better?
Cthulhu_ 2y ago
> But that struggling was ultimately necessary to really learn the concepts. This is what isn't explained or understood properly (...I think) to students; on the surface you go to college/uni to learn a subject, but in reality, you "learn to learn". The output that you're asked to submit is just to prove that you can and have learned. But you don't learn to learn by using AI tools. You may learn how to craft stuff that passes muster, gets you a decent grade and eventually a piece of paper, but you haven't learned to learn. Of course, that isn't anything new, loads of people try and game the system, or just "do the work, get the paper". A box ticking exercise instead of something they actually want to learn.
casey2 2y ago
>I'd work on something for an hour before I'd have an epiphany and figure out what was wrong. But that struggling was ultimately necessary to really learn the concepts. This is entirely your opinion. We don't know how the brain learns nor do we know if intelligence can be "taught"
wordofx 2y ago
It’s not a wide spread “problem”. It’s just education lagging behind technology.
borg16 2y ago
here is an idea, curious what others think of this: split the entire coursework into two parts: part 1 - students are prohibited from using AI. Have the exams be on physical papers than digital ones requiring use of laptop/computer. I know this adds burden on corrections and evaluations of these answers, but I think this provides a raw answer to someone's understanding of concepts being taught in the class. part2 - students are allowed, and even encouraged to use LLMs. And they are evaluated based on the overall quality of the answer, keeping in mind that a non zero portion of this was generated using an LLM. Here the credit should be given to the factual correctness of the answer (and if the student is capable of verifying the LLM output). Have the final grade be some form of weighted average of a student's scores in these 2 parts. note: This is a raw thought that just occurred to me while reading this thread, and I have not had the chance to ruminate on it.
yamazakiwi 2y ago
I'm more interested in memory and knowledge retention in general and how AI can assist. How many times have you heard from people that they are doing rote memorization and will "data dump" test information once a course is over. These tools are less to blame than the motivators and systems that are suppose to be engaging students in real learning and the benefits of a struggle. Another problem is there is so much in technology, I just can't remember everything after years of exposure to so many spaces. Not being able to recall information you used to know is frustrating and having AI to remind you of details is very useful. I see it as an amplifying tool, not a replacement for knowledge. I'm sure there are some prolific note taking memory tricksters out there but I'm not one of them. I frequently forget information over time and it's nice to have a tool to remind me of how UDP, RTP, and SIP routing work when I haven't been in the comm or network space for a while.
- nextos 2y ago
  My CS undergrad school used to let students look up documentation during coding exams. Most courses had a 3-5 hour coding challenge where you had to make substantial changes to a course project you had developed. I think this could also be the right response to LLMs. Let students use whatever they want to use, and test true skills and understanding. FWIW, exams testing rote learning without the ability to look up things would have been much easier. It was really stressful to sit down and make major changes to your project to satisfy new unit tests, which often targeted edge cases and big O complexity to crash your code.
  - yamazakiwi 2y ago
    That is a great idea! Most students would find getting their hands dirty in this way more valuable than reading about something from start to end.
    - nextos 2y ago
      Yes, it led to well-rounded learning. But we had too many courses and, overall, I think it was too much. All CS courses had a theoretical exam, some project-based learning, and some coding exam to prevent cheating in the project-based learning part.
deleted 2y ago
[deleted]
tomxor 2y ago
> I think the issue is that it's so tempting to lean on AI. This is not the root cause, it's a side effect. Student's cheat because of anxiety. Anxiety is driven by grades, because grades affect failure. To detect cheating is solving the wrong problem. If most of the grades did not directly affect failure, student's wouldn't be pressured to cheat. Evaluation and grades have two purposes: 1. Determine grade of qualification i.e result of education (sometimes called "summative") 2. Identify weaknesses to aid in and optimise learning (sometimes called "formative") The problem arises when these two are conflated, either by combining them and littering them throughout a course, or when there is an imbalance in the ratio between them i.e too much of #1. Then the pressure to cheat arises, the measure becomes the target, and focus on learning is compromised. This is not a new problem, student's already waste time trying to undermine grades through suboptimal learning activities like "cramming". The funny thing is that everyone already knows how to solve cheating: controlled examination, which is practical to implement for #1, so long as you don't have a disruptive number of exams filling that purpose. This is even done in sci-fi, Spok takes a "memory test" in 2286 on Vulkan as a kind of "final exam" in a controlled environment with challenges from computers - it's still using a combination of proxy knowledge based questions and puzzles, but it doesn't matter, it's a controlled environment. What's needed is a separation and balance between summative and formative grading, then preventing cheating is almost easy, and student's can focus on learning... cheating at tests throughout the course would actually have a negative affect on their final grade, because they would be undermining their own learning by breaking their own REPL. LLMs have only increased the pressure, and this may end up being a positive thing for education.
toxicdevil 2y ago
I once had an algorithms professor who would give us written home assignments and then on the day of submission take a quiz with identical questions. A significant portion of the class did poorly on these quizes despite scoring good on the assignment. I can't even imagine how learning is impacted by the (ab)use of AI.
chipsrafferty 2y ago
This is frequently stated, but is there any evidence that the "epiphany" is actually required for learning?

stv_123 2y ago

Yeah, the concept of "productive struggle" is important to the education process and having a way to short circuit it seems like it leads to worse learning outcomes.

umpalumpaaa 2y ago
I am not sure all humans work the same way though. Some get very very nervous when they begin to struggle. So nervous that they just stop functioning. I felt that during my time in university. I absolutely loved reading and working through dense math text books but the moment there was a time constraint the struggle turned into chaos.
- AlexandrB 2y ago
  > Some get very very nervous when they begin to struggle. So nervous that they just stop functioning. I sympathize, but it's impossible to remove all struggle from life. It's better in the long run to work through this than try to avoid it.

yapyap 2y ago

> I think the issue is that it's so tempting to lean on AI. I remember long nights struggling to implement complex data structures in CS classes. I'd work on something for an hour before I'd have an epiphany and figure out what was wrong. But that struggling was ultimately necessary to really learn the concepts. With AI, I can simply copy/paste my code and say "hey, what's wrong with this code?" and it'll often spot it (nevermind the fact that I can just ask ChatGPT "create a b-tree in C" and it'll do it). That's amazing in a sense, but also hurts the learning process. In the end the willingness to struggle will set apart the truly great Software Engineer from the AI-crutched. Now of course this will most of the time not be rewarded, when a company looks at two people and sees “passable” code from both but one is way more “productive” with it (the AI-crutched engineer) they’ll inititally appreciate this one more. But in the long run they won’t be able to explain the choices made when creating the software, we will see the retraction from this type of coding when the first few companies’ security falls apart like a house of cards due to AI reliance. It’s basically the “instant gratification vs delayed gratification” argument but wrapped in the software dev box.

JohnMakin 2y ago
I don't wholly disagree with this post, but I'd like to add a caveat, observing my own workflow with these tools. I guess I'd qualify to you as someone "AI crutched" but I mostly use it for research and bouncing ideas (or code complete, which I've mentioned before - this is a great use of the tool and I wouldn't consider it a crutch, personally). For instance, "parse this massive log output, and highlight anything interesting you see or any areas that may be a problem, and give me your theories." Lots of times its wrong. Sometimes its right. Sometimes, its response gives me an idea that leads to another direction. It's essentially how I was using google + stack overflow ten years ago - see your list of answers, use your intuition, knowledge, and expertise to find the one most applicable to you, continue. This "crutch" is essentially the same one I've always used, just in different form. I find it pretty good at doing code review for myself before I submit something more formal, to catch any embarrassing or glaringly obvious bugs or incorrect test cases. I would be wary of the dev that refused to use tools out of some principled stand like this, just as I'd be wary of a dev that overly relied on them. There is a balance. Now, if all you know are these tools and the workflow you described, yea, that's probably detrimental to growth.

vunderba 2y ago

I've been calling this out since the rise of ChatGPT: "The real danger lies in their seductive nature - over how tempting it becomes to immediately reach for the LLM to provide an answer, rather than taking a few moments to quietly ponder the problem on your own. By reaching for it to solve any problem at nearly an instinctual level you are completely failing to cultivate an intrinsically valuable skill - that of critical reasoning."

nonethewiser 2y ago
Somewhat agree. I agree in principal - the process of problem solving is the important part. However I think LLMs make you do more of this because of what you can offload to the LLM. You can offload the simpler things. But for the complex questions that cut across multiple domains and have a lot of ambiguity? You're still going to have to sit down and think about it. Maybe once you've broken it into sufficiently smaller problems you can use the LLM. If we're worried about abstract problem solving skills that doesnt really go away with better tools. It goes away when we arent the ones using the tools.
- Planktonne 2y ago
  You can offload the simpler things, but struggling with the simpler things is how you build the skills to handle the more complex ones that you can't hand off. If the simpler thing in question is a task you've already mastered, then you're not losing much by asking an LLM to help you with it. If it's not trivial to you though, then you're missing an opportunity to learn.
  - jplusequalt 2y ago
    Couldn't have said it better myself. The biology of the human brain will not change as a result of these LLMs. We are imperfect and will tend to take the easiest route in most cases. Having an "all powerful" tool that can offload the important work of figuring out tough problems seems like it will lead to a society less capable in solving complex problems.
  - 2y ago

bko 2y ago

When modern search became more available, a lot of people said there's no point of rote memorization as you can just do a Google search. That's more or less accepted today. Whenever we have a new technology there's a response "why do I need to learn X if I can always do Y", and more or less, it has proven true, although not immediately. For instance, I'm not too concerned about my child's ability to write very legibly (most writing is done on computers), spell very well (spell check keeps us professional), reading a map to get around (GPS), etc Not that these aren't noble things or worth doing, but they won't impact your life too much if you're not interest in penmanship, spelling, or cartography. I believe LLMs are different (I am still stuck in the moral panic phase), but I think my children will have a different perspective (similar to how I feel about memorizing poetry and languages without garbage collection). So how do I answer my child when he asks "Why should I learn to do X if I can just ask an LLM and it will do it better than me"

nyeah 2y ago
Spell check isn't really adequate. You get a page full of correctly spelled words, but they're the wrong words.
- walthamstow 2y ago
  Try being British, often they're not correctly spelt words at all.
  - deleted 2y ago
    [deleted]
delusional 2y ago
"More or less" is doing a lot of work there. School, at least where I am, still spends the first year getting children to memorize the order of the numbers from 1-20 and if there's an even or odd number of a thing on a picture. Do you google if 5 is less than 6 or do you just memorize that? If you believe that creativity is not based on a foundation of memorization and experience (which is just memorization) you need to reflect on the connection between those.

taftster 2y ago

I don't think asking "what's wrong with my code" hurts the learning process. In fact, I would argue it helps it. I don't think you learn when you have reached your frustration point and you just want the dang assignment completed. But before reaching that point, if you had a tutor or assistant that you could ask, "hey, I'm just not seeing my mistake, do you have ideas" goes a long way to foster learning. ChatGPT, used in this way, can be extremely valuable and can definitely unlock learning in new ways which we probably even haven't seen yet. That being said, I agree with you, if you just ask ChatGPT to write a b-tree implementation from scratch, then you have not learned anything. So like all things in academia, AI can be used to foster education or cheat around it. There's been examples of these "cheats" far before ChatGPT or Google existed.

SoftTalker 2y ago
No I think the struggle is essential. If you can just ask a tutor (real or electronic) what is wrong with your code, you stop thinking and become dependent on that. Learning to think your way through a roadblock that seems like a showstopper is huge. It's sort of the mental analog of weight training. The only way to get better at weightlifting is to actually lift weight.
- taftster 2y ago
  If I were to go and try to bench 300lbs, I would absolutely need a spotter to rescue me. Taking on more weight than I can possibly achieve is a setup for failure. Sure, I should probably practice benching 150lbs. That would be a good challenge for me and I would benefit from that experience. But 300lbs would crush me.
  - theamk 2y ago
    Sadly, ChatGPT is like a spotter that takes over at the smallest hint of struggle. Yes, you are not going to get crushed, but you won't get any workout done either. You really want start with a smaller weight, and increment it in steps as you progress. You know, like a class or something. And when you do those exercises, you really want to be lifting those weights yourself, and not rely on spotter for every rep.

Maskawanian 2y ago

Agreed, the only thing that is certain is that they are cheating themselves. While it can be useful to use LLMs as a tutor if you're stuck. The moment that you use it to provide a solution, you stop learning and the tool becomes a required stepping stone.

hobo_in_library 2y ago

The challenge is that while LLMs do not know everything, they are likely to know everything that's needed for your undergraduate education. So if you use them at that level you may learn the concepts at hand, but you won't learn _how to struggle_ to come up with novel answers. Then later in life when you actually hit problem domains that the LLM wasn't trained in, you'll not have learned the thinking patterns needed to persist and solve those problems. Is that necessarily a bad thing? It's mixed: - You lower the bar for entry for a certain class of roles, making labor cheaper and problems easier to solve at that level. - For more senior roles that are intrinsically solving problems without answers written in a book or a blog post somewhere, you need to be selective about how you evaluate the people who are ready to take on that role. It's like taking the college weed out classes and shifting those to people in the middle of their career. Individuals who can't make the cut will find themselves stagnating in their roles (but it'll also be easier for them to switch fields). Those who can meet the bar might struggle but can do well. Business will also have to come up with better ways to evaluate candidates. A resume that says "Graduated with a degree in X" will provide less of a signal than it did in the past

psygn89 2y ago

Agreed, the struggle often leads us to poke and prod an issue from many angles until things finally click. It lets us think critically. In that journey you might've learned other related concepts which further solidifies your understanding. But when the answer flows out of thin air right in front of you with AI, you get the "oh duh" or "that makes sense" moments and not the "a-ha" moment that ultimately sticks with you. Now does everything need an "a-ha" moment? No. However, I think core concepts and fundamentals need those "a-ha" moments to build a solid and in-depth foundation of understanding to build upon.

taftster 2y ago
Absolutely this. AI can help reveal solutions that weren't seen. An a-ha moment can be as instrumental to learning as the struggle that came before. Academia needs to embrace this concept and not try to fight it. AI is here, it's real, it's going to be used. Let's teach our students how to benefit from its (ethical) use.
porridgeraisin 2y ago
Yep. People love to cut down this argument by saying that a few decades ago, people said the same thing about calculators. But that was a problem too! People losing a large portion of their mental math faculty is definitely a problem. If mental math was required daily, we wouldn't see such obvious BS numbers in every kind of reporting(media/corporate/tech benchmarks) that people don't bat an eye at. How much the problem is _worth_ though, is what matters for adoption of these kinds of tech. Clearly, the problem above wasn't worth much. We now have to wait and see how much the "did not learn through cuts and scratches" problem is worth.

dyauspitr 2y ago

I’m pretty sure you can assume close to 100% of students are using LLMs to do their homework.

ryandrake 2y ago
And if you're that one person out of 100,000 who is not using LLMs to do their homework, you are at a significant disadvantage on the grading curve.
- DontchaKnowit 2y ago
  Maybe, but piss on that, who needs good grades? Youll learn a hell of a lot better
- Eavolution 2y ago
  Currently in university and my experience is that it heavily depends on the module. For a lot your statement's probably accurate, however for others it really isn't. For example we have a microprocessors module which is programming for an rp2040 in c, but also manually setting up interrupt handlers etc in assembly. All of the LLM's are completely useless for it, they tell you that the rp2040 works in ways it just doesn't and are actively unhelpful with the misinformation. The only students who can do well in that module are the ones that understand the material well and go to the datasheet and documentation instead of an LLM.
- dysoco 2y ago
  My university solves this quite easily. There is no graded homework, the coursework is there only as a guide and practice for the exams. So you can absolutely use LLMs to help you with the exercises or to help understand something, however if you blindly get answers you will only be fooling yourself as you won't be able to pass the exams.
  - dyauspitr 2y ago
    That’s how most schooling has already been in a lot of South and East Asia. If you don’t do your homework, you get punished in other ways, but it doesn’t have any impact on the overall grade, the grade solely depends on the final exam.

andai 2y ago

I spent much of the past year at public libraries, and I heard the word ChatGPT approximately once per minute, in surround sound. Always from young people, and usually in a hushed tone...

moltar 2y ago

I think it’s finally time to just stop the homework. All school work must be done within the walls of the school. What are we teaching our children? It’s ok to do more work at home? There are countries that have no homework and they do just fine.

jplusequalt 2y ago
Homework helps reinforce the material learned in class. It's already a problem where there is too much material to be fit into a single class period. Trying to cram in enough time for homework will only make that problem worse.
- moltar 2y ago
  Can do the work the next day to reinforce. As I said there are countries without homework and they seem to do ok. So it’s not mandatory by any means.
  - jplusequalt 2y ago
    >Can do the work the next day to reinforce. Keeping the curriculum fixed, there's already barely enough time to cover everything. Cutting the amount of lectures in half to make room for in-class homework time does not fix this fundamental problem.
    - DontchaKnowit 2y ago
      Just make lecture times longer.
      - umbra07 2y ago
        students already don't pay attention in lecture: * due to either learning/concentration issues * the fact that most lecturers are boring, dull, and unengaging * and oftentimes you can learn better from other sources making lecture longer doesn't fix a single one of these issues. it just makes students learn even less.

srveale 2y ago

IMO it's so easy to ChatGPT your homework that the whole education model needs to flip on its head. Some teachers already do something like this, it's called the "Flipped classroom" approach. Basically, a student's marks depend mostly (only?) on what they can do in a setting where AI is verifiably unavailable. It means less class time for instruction, but students have a tutor in their pocket anyway. I've also talked with a bunch of teachers and a couple admins about this. They agree it's a huge problem. By the same token, they are using AI to create their lesson plans and assignments! Not fully of course, they edit the output using their expertise. But it's funny to imagine AI completing an AI assignment with the humans just along for the ride. The point is, if you actually want to know what a student is capable of, you need to watch them doing it. Assigning homework has lost all meaning.

vonneumannstan 2y ago
>Not fully of course, they edit the output using their expertise Surely this is sarcasm, but really your average schoolteacher is now a C student Education Major.
- srveale 2y ago
  I was talking about people I know and talk with, mostly friends and family, who are smart, hard working, and their students are lucky to have them.
hackyhacky 2y ago
> it's called the "Flipped classroom" approach. Flipped classroom is just having the students give lectures, instead of the teacher. > Basically, a student's marks depend mostly (only?) on what they can do in a setting where AI is verifiably unavailable. This is called "proctored exams" and it's been pretty common in universities for a few centuries. None of this addresses the real issue, which is whether teachers should be preventing students from using AIs.
- srveale 2y ago
  > Flipped classroom is just having the students give lectures, instead of the teacher. Not quite. Flipped classroom means more instruction outside of class time and less homework. > This is called "proctored exams" and it's been pretty common in universities for a few centuries. None of this addresses the real issue Proctored exams is part of it. In-class assignments is another. Asynchronous instruction is another. And yes, it addresses the issue. Students can use AI however they see fit, to learn or to accomplish tasks or whatever, but for actual assessment of ability they cannot use AI. And it leaves the door open for "open-book" exams where the use of AI is allowed, just like a calculator and textbook/cheat-sheet is allowed for some exams. https://en.wikipedia.org/wiki/Flipped_classroom https://en.wikipedia.org/wiki/Flipped_classroom

chalst 2y ago

Students who do that risk submitting assignments that show they don’t understand the course so far.

0xffff2 2y ago

>For many students, it's literally "let me paste the assignment into ChatGPT and see what it spits out, change a few words and submit that". Does that actually work? I'm long past having easy access to college programming assignments, but based on my limited interaction with ChatGPT I would be absolutely shocked if it produced output that was even coherent, much less working code given such an approach.

StefanBatory 2y ago
I have some subjects, at Masters - that are solvable by one prompt. One. Quality of CS/Software Engineering programs vary that much.
- porridgeraisin 2y ago
  Yeah. On the other hand, "implement boruvkas MST algorithm in cuda such that only the while(numcomponents > 1) loop runs on the CPU, and everything else runs in the gpu. Memcpy everything onto the gpu first and only transfer back the count each iteration/keep it in pinned memory" It never gets it right, even after many reattempts in cursor. And even if it gets it right, it doesn't do the parallelization effectively enough - it's a hard problem to parallelize.
bongodongobob 2y ago
Why are you asking? Go try it. And yes, depending on the task, it does.
- 0xffff2 2y ago
  As I said, I'm not a student, so I don't have access to a homework assignment to paste in. Ironically I have pretty much everything I ever submitted for my undergrad, but it seems like I absolutely never archived the assignments for some reason.
  - bongodongobob 2y ago
    I was able to get ~80% one shots on Advent of Code with 4o up to about day 12 iirc.

ryandrake 2y ago

I think teachers also need to reconsider how they are measuring mastery in the subject. LLMs exist. There is no putting the cat back into the bag. If your 1980s way to measure a student's mastery of a subject can be fooled by an LLM, then how effective is that measurement in 2020+? Maybe we need to stop using essays as a way to tell if the student has learned the material. Don't ask me what the solution is. Maybe your product does it. If I knew, I'd be making a fortune selling it to universities.

teekert 2y ago

Students do something akin to vibe coding I guess. It may seem impressive at first glance but if anything breaks you are so, so lost. Maybe that’s it, break the student’s code, see how they fix it. The vibe coding student is easily separate from the real one (of course this real coder can also use AI, just not yoloing it). I guess you can apply similar mechanics to reports. Some deeper questions and you will know if the report was self written or if an AI did it.

bboygravity 2y ago

I don't get this reasoning. Without LLMs I would learn how to write sub-optimal code that is somewhat functional. With LLMs instantly see "how it's done" for my exact problem case which makes me learn way faster. On top of that it always makes dumb mistakes which forces you to actually understand what it's spitting out to get it to work properly. Again: that helps with learning. The fact that you can ask it for a solution for exactly the context you're interested in is amazing and traditional learning doesn't come close in terms of efficiency IMO.

dingnuts 2y ago
> With LLMs instantly see "how it's done" for my exact problem case which makes me learn way faster. No, you see a plausible set of tokens that appear similar to how it's done, and as a beginner, you're not able to tell the difference between a good example and something that is subtly wrong. So you learn something, but it's wrong. You internalize it. Later, it comes back to bite you. But OpenAI keeps the money for the tokens. You pay whether the LLM is right or not. Sam likes that.
- Spivak 2y ago
  This makes for a good sound bite but it's just not true. The use case of "show me what is a customary solution to <problem>" plays exactly into LLMs strength as a funny kind of search engine. I used to (and still do) search public code for this use case to get a sense of the style and idioms common in a new language/library and the plausible set of tokens is doing exactly that.
- margalabargala 2y ago
  > So you learn something, [...] You internalize it. Or they don't.
layer8 2y ago
It’s more like looking up the solution to the math problem you’re supposed to solve on your own. It can be helpful in some situations, but in general you don’t learn the problem-solving skills if you don’t do it yourself.

deleted 2y ago

[deleted]

victorbjorklund 2y ago

In one way I'm glad I learned to code before LLM:s. It would be so hard to push through the learning now when you are just a click away from buildning the app with AI...

sally_glance 2y ago

I think this is a structural issue. Universities right now are trying to justify their existence - universities of the past used to be sites of innovation. Using ChatGPT doesn't dumb down your students. Not knowing how it works and where to use it does. Don't do silly textbook challenges for exams anymore - reestablish a culture of scientific innovation!

acbart 2y ago
Incorrect. Fundamentals must be taught in order to provide the context for the more challenging open-ended activities. Memorization is the base of knowledge, a starting point. Cheating (whether through an LLM or hiring someone or whatever) skips the journey. You can't just take them through the exciting routes, sometimes they have to go through the boring tedious repetitive stuff because that's how human brains learn. Learning is, literally, a stressful process on the brain. Students try to avoid it, but that's not good for them. At least in the introductory core classes.
- sally_glance 2y ago
  I guess I should have phrased it differently - what I meant was just stop testing the tedious stuff, make it clear to students that learning the fundamentals is expected. Then examine them on hard exploratory problems which require the fundamentals.
nixpulvis 2y ago
You claim using AI tools doesn't dumb you down, but it very well could and is. Take the calculator for example, I'm overly dependent on it. I'm slower to perform arithmetic than I would have been without it. But knowing how to use one allows me to do more complex math more quickly. So I'm "dumber" in one way and "smarter" in others. AI could be the same... except our education system doesn't seem ready for it. We still learn arithmetic, even if we later rely in tools to do it. Right now teachers don't know how to teach so that AI doesn't trivialize things. You need to know how to do things so you know when the AI is lying to you.
- 2y ago

aj7 2y ago

I’m a physicist. I can align and maximize ANY laser. I don’t even think when doing this task. Long hours of struggle, 50 years ago. Without struggle there is nothing. You can bullshit your way in. But you will be ejected.

ketzo 2y ago
barely related to your point but “I can align and maximize ANY laser” is such an incredibly specific flex, I love it
- ramraj07 2y ago
  Especially because it's not a skill everyone gets just because they practice. I know because I tried for years lol.
marksbrown 2y ago
A master blacksmith can shoe a horse an' all. Laser alignment is also a solved problem with a machine. Just because something can be done by hand does not mean it has any intrinsic value.

whatever1 2y ago

The counter argument is that now you can skip boilerplate code and focus on the overall design and the few points that brainpower is really needed. The amount of visualizations that i have made after chat gpt was released has increased exponentially. I loath looking the documentation again and again to make a slightly non standard graph. Now all of the friction is gone! Graphs and visuals are everywhere in my code!

hansvm 2y ago
> focus on [...] the few points that brainpower is really needed The person you're responding to is talking about it from an educational perspective though. If your fundamentals aren't solid, you won't know that exponentially smoothed reservoir sampling backed by a splay tree is optimal for your problem, and ChatGPT has no clue either. Trying things, struggling, and failing is crucial to efficient learning. Not to mention, you need enough brain power or expertise to know when it's bullshitting you. Just today it was telling me that a packed array was better than my proposed solution, confidently explaining why, and not once saying anything correct. No prompt changes could fix it (whether restarting or replying), and anyone who tried to use less brainpower there would be up a creek when their solution sucked. Mind you, I use LLMs a lot, including for code-adjacent tasks and occasionally for code itself. It's a neat tool. It has its place though, and it must be used correctly.

musicale 2y ago

> “how much are students using AI to cheat?” That’s hard to answer "It is difficult to get a man to understand something, when his salary depends on his not understanding it!"

enjo 2y ago

> it's literally "let me paste the assignment into ChatGPT and see what it spits out, change a few words and submit that". My wife is an accounting professor. For many years her battle was with students using Chegg and the like. They would submit roughly correct answers but because she would rotate the underlying numbers they would always be wrong in a provably cheating way. This made up 5-8% of her students. Now she receives a parade of absolutely insane answers to questions from a much larger proportion of her students (she is working on some research around this but it's definitely more than 30%). When she asks students to recreate how they got to these pretty wild answers they never have any ability to articulate what happened. They are simply throwing her questions at LLMs and submitting the output. It's not great.

DSingularity 2y ago
This is now reality -- fighting to change the students is a losing battle. Besides in terms of normalizing grade distributions this is not that complicated to solve. Target the cheaters with pop quizzes. Prof can randomly choose 3 questions from assignments. If students cant get enough marks on 2/3 of them they are dealt a huge penalty. Students that actually work through the problems will have no problems with scoring enough marks on 2/3 of the questions. Students that lean irresponsibly on LLMs will lose their marks.
- rrr_oh_man 2y ago
  Maybe we'll revert to Soviet bilet-style oral exams...
  - skandinaff 2y ago
    But they do that already in Danish universities, and I have to admit, those are very effective.
- cellularmitosis 2y ago
  Why not just grade solely based on live performance? (quizzes and tests) Homework would still be assigned as a learning tool, but has no impact on your grade.

ugh123 2y ago

>I built a popular product that helps teachers with this problem. Does your product help teachers detect cheating? Because I hear none of them are accurate, with many false positives and ruined academic careers. Are you saying yours is better?

Cthulhu_ 2y ago

> But that struggling was ultimately necessary to really learn the concepts. This is what isn't explained or understood properly (...I think) to students; on the surface you go to college/uni to learn a subject, but in reality, you "learn to learn". The output that you're asked to submit is just to prove that you can and have learned. But you don't learn to learn by using AI tools. You may learn how to craft stuff that passes muster, gets you a decent grade and eventually a piece of paper, but you haven't learned to learn. Of course, that isn't anything new, loads of people try and game the system, or just "do the work, get the paper". A box ticking exercise instead of something they actually want to learn.

casey2 2y ago

>I'd work on something for an hour before I'd have an epiphany and figure out what was wrong. But that struggling was ultimately necessary to really learn the concepts. This is entirely your opinion. We don't know how the brain learns nor do we know if intelligence can be "taught"

wordofx 2y ago

It’s not a wide spread “problem”. It’s just education lagging behind technology.

borg16 2y ago

here is an idea, curious what others think of this: split the entire coursework into two parts: part 1 - students are prohibited from using AI. Have the exams be on physical papers than digital ones requiring use of laptop/computer. I know this adds burden on corrections and evaluations of these answers, but I think this provides a raw answer to someone's understanding of concepts being taught in the class. part2 - students are allowed, and even encouraged to use LLMs. And they are evaluated based on the overall quality of the answer, keeping in mind that a non zero portion of this was generated using an LLM. Here the credit should be given to the factual correctness of the answer (and if the student is capable of verifying the LLM output). Have the final grade be some form of weighted average of a student's scores in these 2 parts. note: This is a raw thought that just occurred to me while reading this thread, and I have not had the chance to ruminate on it.

yamazakiwi 2y ago

I'm more interested in memory and knowledge retention in general and how AI can assist. How many times have you heard from people that they are doing rote memorization and will "data dump" test information once a course is over. These tools are less to blame than the motivators and systems that are suppose to be engaging students in real learning and the benefits of a struggle. Another problem is there is so much in technology, I just can't remember everything after years of exposure to so many spaces. Not being able to recall information you used to know is frustrating and having AI to remind you of details is very useful. I see it as an amplifying tool, not a replacement for knowledge. I'm sure there are some prolific note taking memory tricksters out there but I'm not one of them. I frequently forget information over time and it's nice to have a tool to remind me of how UDP, RTP, and SIP routing work when I haven't been in the comm or network space for a while.

nextos 2y ago
My CS undergrad school used to let students look up documentation during coding exams. Most courses had a 3-5 hour coding challenge where you had to make substantial changes to a course project you had developed. I think this could also be the right response to LLMs. Let students use whatever they want to use, and test true skills and understanding. FWIW, exams testing rote learning without the ability to look up things would have been much easier. It was really stressful to sit down and make major changes to your project to satisfy new unit tests, which often targeted edge cases and big O complexity to crash your code.
- yamazakiwi 2y ago
  That is a great idea! Most students would find getting their hands dirty in this way more valuable than reading about something from start to end.
  - nextos 2y ago
    Yes, it led to well-rounded learning. But we had too many courses and, overall, I think it was too much. All CS courses had a theoretical exam, some project-based learning, and some coding exam to prevent cheating in the project-based learning part.

deleted 2y ago

[deleted]

tomxor 2y ago

> I think the issue is that it's so tempting to lean on AI. This is not the root cause, it's a side effect. Student's cheat because of anxiety. Anxiety is driven by grades, because grades affect failure. To detect cheating is solving the wrong problem. If most of the grades did not directly affect failure, student's wouldn't be pressured to cheat. Evaluation and grades have two purposes: 1. Determine grade of qualification i.e result of education (sometimes called "summative") 2. Identify weaknesses to aid in and optimise learning (sometimes called "formative") The problem arises when these two are conflated, either by combining them and littering them throughout a course, or when there is an imbalance in the ratio between them i.e too much of #1. Then the pressure to cheat arises, the measure becomes the target, and focus on learning is compromised. This is not a new problem, student's already waste time trying to undermine grades through suboptimal learning activities like "cramming". The funny thing is that everyone already knows how to solve cheating: controlled examination, which is practical to implement for #1, so long as you don't have a disruptive number of exams filling that purpose. This is even done in sci-fi, Spok takes a "memory test" in 2286 on Vulkan as a kind of "final exam" in a controlled environment with challenges from computers - it's still using a combination of proxy knowledge based questions and puzzles, but it doesn't matter, it's a controlled environment. What's needed is a separation and balance between summative and formative grading, then preventing cheating is almost easy, and student's can focus on learning... cheating at tests throughout the course would actually have a negative affect on their final grade, because they would be undermining their own learning by breaking their own REPL. LLMs have only increased the pressure, and this may end up being a positive thing for education.

toxicdevil 2y ago

I once had an algorithms professor who would give us written home assignments and then on the day of submission take a quiz with identical questions. A significant portion of the class did poorly on these quizes despite scoring good on the assignment. I can't even imagine how learning is impacted by the (ab)use of AI.

chipsrafferty 2y ago

This is frequently stated, but is there any evidence that the "epiphany" is actually required for learning?