15 ms·
Student perceptions of AI coding assistants in learning
- bgwalter 11mo agoIt is notable that so many publications try to salvage "AI" ("need for new pedagogical approaches that integrate AI effectively") rather than ditch "AI" completely. The world worked perfectly before 2023, there is no need to outsource information retrieval or thinking.
- Wowfunhappy 11mo agoThe world worked perfectly before 1982, there is no need for the internet. (…I actually kind of think this. "Kind of" being the key word.)
- awesome_dude 11mo agoGod no. Speaking as someone that communicates primarily through text (high likelihood of Autism) the internet was the first chance a lot of us had to ... speak.. and be heard
- deadbabe 11mo agoThat’s not a problem that generalizes to the broader population. We don’t really need internet.
- awesome_dude 11mo agoScrew the broader population I can speak now dammit!!!!!
- danaris 11mo agoIn other words "disabled people can suck it, because I don't care about their lives or experiences"? We often fall short, but as a society we do try to make sure we're accommodating disabled people when we make big changes in our systems.
- awesome_dude 11mo agoJust FTR - I read them as being facetious
- danaris 11mo agoPoe's Law applies. There are far, far too many people who genuinely think disabled people should just disappear or die for it to be "safe" to be facetious about that without a clear sarcasm indicator.
- cindyllm 11mo ago[dead]
- fn-mote 11mo agoI disagree. People have a need to be heard and understood. That’s half of what we are doing here posting. Many (“not disabled”) people don’t fit in with their local peer group / society. The internet gave them a way to connect with other like-minded individuals. Do I need to give examples? Let’s say: struggling with a rare disease.
- Wowfunhappy 11mo agoBut on the other end you have genocide triggered by Facebook. You can't speak if you're dead. Perhaps some of that violence would have happened anyway. I don't know how it all nets out.
- sukhdeepprashit 11mo ago[flagged]
- manmal 11mo agoNot much has changed, only people get diagnosed now. I think GP makes actually a good point that, with all its downsides, there are also net positive upsides to the internet.
- bad_haircut72 11mo agothere are upsides but I dont know if its net upside. In this particular example, communicating by text - letter writing has existed for millenia and has arguably degraded considerably in this age of instant messaging
- awesome_dude 11mo agoSorry, i know it's a bit "flavour of the month" but I mentioned it because I have a difficulty communicating face to face, which is common amongst a certain group of people, and I figured that mentioning it would help people understand my thinking.
- bad_haircut72 11mo agoWhy couldn't you write letters instead of texting?
- awesome_dude 11mo agoI didn't have your mailing address :P
- boca_honey 11mo agoAh yes, the perfect world we had when governments could get away with anything because the press was not enough to showcase their attrocities. A beautiful, perfect world, with rubella and a global population living in extreme poverty close to 50% (compared to today's 10%). I see this mentality almost exclusively in americans and/or anglo people in general, it's incredible... if you're not that, I guess you're just too young or completely isolated from reality and I wish you the best in the ongoing western collapse. (... I actually wish you're joking and I didn't catch it, though).
- daseiner1 11mo agolast sentence in your first paragraph has nothing to do with the current state of the internet and certainly not AI. first sentence? turns out governments can still get away with pretty much anything and propaganda is easier than ever.
- pessimizer 11mo ago> propaganda is easier than ever. It is so much harder now. There are people who are willfully ignorant now, almost proud to be; snooty about it. But it's impossible for governments and institutions to lie like they used to be able to. People are trading primary source documents online within the day. It's why the popularity of long-ruling institutional parties is dropping everywhere, and why the measures to stop people from communicating and to monitor what they're saying are becoming more and more draconian and desperate.
- daseiner1 11mo agobeyond irony that you pose as some tech optimist while also mentioning “western collapse” and then speak about a uniquely American pessimism, a nation that is presently under the thumb of a government that does not respect the rule of law and actively manipulates capital/big business. and you cannot simply hand-wave away the massive acceleration of the surveillance state and characterize it as a tool of the “institutional parties”
- rezz 11mo agoWhy stop there? We could do long division before the calculator and hand write before the typewriter.
- maplethorpe 11mo agoI do wonder if the calculator would have been as successful if it regularly delivered wrong answers.
- Bootvis 11mo agoIt does if you’re a clumsy operator and those are not rare.
- pfortuny 11mo agoYes, but the machine itself is deterministic and logically sound.
- ramesh31 11mo ago>Yes, but the machine itself is deterministic and logically sound. Because arithmetic itself, by definition, is. Human language is not. Which is why being able to talk to our computers in natural language (and have them understand us and talk back) now is nothing short of science fiction come true.
- maplethorpe 11mo agoEven worse is if it's in the other room and your fingers can't reach the keys. It delivers no answers at all!
- Bootvis 11mo agoMy point is, needing to use something with care doesn't prevent it becoming from wildly successful. LLM's are wrong way more often but are also more versatile than a calculator.
- deleted 11mo ago[deleted]
- ako 11mo agoWhere is this perfect world you’re speaking of? Surely not the one we’re living in…
- mgraczyk 11mo agoI'm a lot more productive than I was in 2023 and I've been coding full time since 2012
- dcre 11mo agoInteresting to see quotes but note N=20 and the methodology doesn’t seem all that rigorous. I didn’t see anything that wasn’t exactly what you would expect to hear.
- felipeerias 11mo agoIn these studies, the qualitative data is often a lot more informative than the quantitative. Understanding how concrete people navigate a domain and noting the common points between them can be illuminating. Trying to calculate a generalisable statistical result from them… probably not so much.
- taurath 11mo agoI am not a student and I wonder often whether we fill in memorization for the idea of learning, as though it’s somehow more valuable to be able to write valid syntax from memory on a blank file than it is to know and practice the broader strokes of abstractions, operators, readability and core concepts which make up good software craftsmanship. Sometimes I’m doing something in a new to me language, using an LLM to give me a head start on structure and to ask questions about conventions and syntax, and wondering to myself how much I’m missing had I started just by reading the first half of a book on the language. I think I probably would take a lot longer to do anything useful, but I’d probably also have a deeper understanding of what I know and don’t know. But then, I can just as easily discover those fundamental concepts to a language via the right prompt. So am I learning? Am I somehow fooling myself? How?
- jjmarr 11mo agoBecause not everyone can truly be great at their craft, but everyone can memorize syntax. Schools compromise their curriculum so that every student has a chance in the interests of fairness.
- tehjoker 11mo agoYou have to know the basics to build higher level knowledge and skills. What’s the use of high level book learning without the ability to operationalize it
- thomasahle 11mo agoI'm not sure we really know how much of learning is memorization. As we memorize more stuff, we find patterns to compress it and memorize more efficiently.
- raincole 11mo agoSounds awfully like machine learning, doesn't it?
- 11mo ago
- awongh 11mo ago> Automatic creation of an initial billboard: Upon starting the program, a predefined list of movies currently showing must be automatically generated, including their details (title, genre, duration, and showtimes). I would say that these results might be relevant for a university CS program setting, but I would make the distinction between this and actually learning to program. The context of this task is definitely a very contrived "Let's learn OOP" assignment that, for example, just tires to cram in class inheritance without really justifying it's use in the software that's being built. It's a lazy kind of curriculum building that doesn't actually tell the students about OOP. In that sense it's no wonder that AI is not that helpful in the context of the assignment and learning. I wouldn't chalk this up to "AI doesn't help you learn". I would put this in the category of, in an overly academic assignment with contrived goals, AI doesn't help the student accomplish the goals of the course. That conclusion could be equally applied to French literature 102. And that's very different from whether or not an AI coding assistant can help you learn to code or not. (I'm actually not sure if it can, but I think this study doesn't say anything new).
- deleted 11mo ago[deleted]
- borski 11mo agoCrazy idea but: what if we built an AI pair programmer that actually pair programmed? That is, sometimes it was the driver and you navigated, pretty much as it is today, but sometimes you drive and it navigates. I surmise that would help people learn to code better.
- manmal 11mo agoLLMs with their current context size suck at navigating in larger codebases. They are better left in the driver seat for now.
- calepayson 11mo ago> Our findings reveal that students perceived AI tools as helpful for grasping code concepts and boosting their confidence during the initial development phase. However, a noticeable difficulty emerged when students were asked to work un-aided, pointing to potential over reliance and gaps in foundational knowledge transfer. As someone studying CS/ML this is dead on but I don't think the side-effects of this are discussed enough. Frankly, cheating has never been more incentivized and it's breaking the higher education system (at least that's my experience, things might be different at the top tier schools). Just about every STEM class I've taken has had some kind of curve. Sometimes individual assignments are curved, sometimes the final grade, sometimes the curve isn't a curve but some sort of extra credit. Ideally it should be feasible to score 100% in a class but I think this actually takes a shocking amount of resources. In reality, professors have research or jobs to attend to and same with the students. Ideally there are sections and office hours and the professor is deeply conscious of giving out assignments that faithfully represent what students might be tested on. But often this isn't the case. The school can only afford two hours of TA time a week, the professors have obligations to research and work, the students have the same. And so historically the curve has been there to make up for the discrepancy between ideals and reality. It's there to make sure that great students get the grades that they deserve. LLMs have turned the curve on its head. When cheating was hard the curve was largely successful. The great students got great grades, the good students got good grades, those that were struggling usually managed a C+/B-, and those that were checked out or not putting in the time failed. The folks who cheated tended to be the struggling students but, because cheating wasn't that effective, maybe they went from a failing grade to just passing the class. A classic example is sneaking identities into a calculus test. Sure it helps if you don't know the identities but not knowing the identities is a great sign that you didn't practice enough. Without that practice they still tend to do poorly on the test. But now cheating is easy and, I think it should change the way we look at grades. This semester, not one of my classes is curved because there is always someone who gets a 100%. Coincidentally, that person is never who you would expect. The students who attend every class, ask questions, go to office hours, and do their assignments without LLMs tend to score in B+/A- range on tests and quizzes. The folks who set the curve on those assignments tend to only show up for tests and quizzes and then sit in the far back corners when they do. Just about every test I take now, there's a mad competition for those back desks. Some classes people just dispense with the desk and take a chair to the back of the room. Every one of the great students I know is murdering themselves to try to stay in the B+/A- range. A common refrain when people talk about this is "cheaters only cheat themselves" and while I think has historically been mostly true, I think it's bullshit now. Cheating is just too easy, the folks who care are losing the arms race. My most impressive peers are struggling to get past the first round of interviews. Meanwhile, the folks who don't show up to class and casually get perfect scores are also getting perfect scores on the online assessments. Almost all the competent people I know are getting squeezed out of the pipeline before they can compete on level-footing. We've created a system that massively incentivizes cheating and then invented the ultimate cheating tool. A 4.0 and a good score on an online assessment used to be a great signal that someone was competent. I think these next few years, until universities and hiring teams adapt to LLMs, we're going to start seeing perfect scores as a red flag.
- ghm2180 11mo agoI wonder when will there be something more rigorous on what works clearing house https://ies.ed.gov/ncee/WWC/Search/Products?searchTerm=AI&&&&&&&&interventionId=&publicationDate=undefined https://ies.ed.gov/ncee/WWC/Search/Products?searchTerm=AI&&&... I am actually hoping someone there studies such interventions the way they did with CMU's intelligent tutor — which if I recall correctly did not have net strong evidence in its favors as far as educational outcomes per the reports in WWC — given the fall in grade level scores in math and reading since 2015/16 across multiple grades in middle school. It is vital to know if any of these things help kids succeed.
- bombdailer 11mo agoKnowledge not earned is not gained.
- bequanna 11mo agoWell said. I’ve often been able to trick myself into thinking I’ve learned something, especially if it is somewhat intuitive. But unless I practically apply what I learned, my retention is quite low.
- insane_dreamer 11mo ago> Our findings reveal that students perceived AI tools as helpful for grasping code concepts and boosting their confidence during the initial development phase. However, a noticeable difficulty emerged when students were asked to work unaided, pointing to potential overreliance and gaps in foundational knowledge transfer. This is basically what would be expected. However n=20 is too small. This needs to be replicated with x10 the n.
- whatever1 11mo agoI have access to so many videos and even video games that teach me exactly how to perform as a world class athlete. If I don’t exercise, will I ever become one?
- nairboon 11mo agoThat analogy sounds good on the surface, but breaks down quickly. In the context of sport/athletics it's your body doing the work, which is difficult to change without exercise (even with steroids). But for knowledge work, as long as you have access to the llm machine, you can quickly pretend it's your work. For students it can be very deceiving, they type their ideas on a device, one button fixes their spelling, another button "fixes" their wording. The line is very blurry, literally the divider in the UI between these two buttons.
- avgDev 11mo agoThis answer is obvious, however, video games/simulators can indicate if someone has potential. Sim racing is a good example of this, there have been several drivers who went pro. Sims are used by racing teams and people like Max Verstappen, Lando Norris use sims heavily to improve their race craft. I do a bit of sim racing and raced against pro drivers. Athletic ability is a huge part but nearly every sport has a chess component. Additionally, ability to perform under pressure and dealing with stress/anxiety is another part of sports.
- pointlessone 11mo agoGood for you it’s obvious for you. But there are way too many people believing they can fight a bear/moose and win.
- mgraczyk 11mo agoThe sad reality is that this is probably not a solvable problem. AI will improve more rapidly than the education system can adapt. Within a few years it won't make sense for people to learn how to write actual code, and it won't be clear until then which skills are actually useful to learn. My recommendation would be to encourage students to ask the LLM to quiz and tutor them, but ultimately I think most students will learn a lot less than say 5 years ago while the top 5% or so will learn a lot more
- andrei_says_ 11mo ago> Within a few years it won't make sense for people to learn how to write actual code Why? Because LLMs are capable of sometimes working snippets of usually completely unmaintainable code?
- mgraczyk 11mo ago[flagged]
- abtinf 11mo agoIt would seem relevant to disclose you work at Anthropic.
- meowface 11mo agoPerhaps. They're still 100% right.
- Madmallard 11mo agoNot at all whatsoever
- ethmarks 11mo agoYou can still argue that LLMs won't replace human programmers without downplaying their capabilities. Modern SOTA LLMs can often produce genuinely impressive code. Full stop. I don't personally believe that LLMs are good enough to replace human developers, but claiming that LLMs are only capable of writing bad code is ridiculous and easily falsifiable.
- adidoit 11mo agoThis sounds like one of the "Ironies of Automation" as Lisain Bainbridge pointed out several years ago. The more prevalent automation is, the worse humans do when that automation is taken away. This will be true for learning now . Ultimately the education system is stuck in a bind. Companies want AI-native workers, students want to work with AI, parents want their kids to be employable. Even if the system wants to ensure that students are taught how to learn and not just a specific curriculum, their stakeholders have to be on board. I think we're shifting to a world where not only will elite status markers like working at places like McKinsey and Google be more valuable but also interview processes will be significantly lengthened because companies will be doing assessments themselves and not trusting credentials from an education system that's suffering from great inflation and automation
- DiscourseFan 11mo agoPerhaps the credentials will change as these academic institutions become more like dinosaurs and other kinds of institutes will arise which give better markers of ability.
- kiba 11mo agoI don't know what AI native folks will look like. To me, it looks like just replacing skilled labors with unskilled labors as opposed to giving humans new skills. AI to me will be valuable when it's helping humans learn and think more strategically and when they're actually teaching humans or helping humans spot contradiction and vetting for reliable information. Fact checking is extremely labor intensive work after all. Or otherwise if AI is so good, just replace humans. Right now, the most legible use of AI is AI slop and misinformation.
- sothatsit 11mo agoThe skilled AI users are the people that use it to help them learn and think problems through in more detail. Unskilled AI users are people who use AI to do their thinking for them, rather than using it as a work partner. This is how people end up producing bad work because they fundamentally don’t understand the work themselves.