14 ms·

Teaching Programming in the Age of ChatGPT

154 points by headalgorithm 3y ago

skepticATX 3y ago
I knew a few people who thought they could cheat their way through a CS degree. They all dropped out or changed majors within a few semesters. It’s generally a problem that solves itself, in my experience. Perhaps a benefit of these tools is that we stop the obsession over cheating, which inconveniences honest students in many ways. Cheating has always occurred, but now we can’t even pretend that it’s preventable.
- NationOfJoe 3y ago
  Would you be able to elaborate on why they dropped out? did they hit a point where cheating wasn't possible? or maybe realized it was pointless and unhelpful once they need to do a job?
  - paxys 3y ago
    In my experience it's always the former. Back in university the surest sign of a good class was that you were allowed to "cheat". The professor encouraged using any and all outside materials for your assignments. You were allowed to work with other students on projects. Exams were all open book. You could use open source code. Such classes always led to the best outcomes for students who were truly interested and engaged, and those who weren't had no way of faking it. On the other hand classes where professors spent more effort on detecting cheating than teaching the material were always duds.
    - danpalmer 3y ago
      Huge +1 to this. I had one lecturer who would essentially rant at us about security vulnerabilities every week for a semester, and then write a paper based on his rants. Some students ignored his rants because they didn't find the subject interesting, did the past papers, and then got to the exam and did terribly, then complained about the fact that "it's all based on his rants". Others found the ranting engaging because it was deep dives on obscure bits of computer security history. My year he spent ~4 weeks going on about Stuxnet, including deep dives into the wider political context. When I got to the paper, one of the few questions we could choose, for 50% of the paper, was just "What was Stuxnet". I wrote pages and pages. Figured out from marking that I got full marks on that one. I always did great in that lecturer's modules despite never taking notes and rarely doing any targeted reading. To show the level of engagement, I once turned up to one of these lectures 10 mins late, with most of the class already there, and the lecturer said "oh I wondered where you guys were, I guess I'll start again". He knew we were the only ones who cared about the class.
      - sitkack 3y ago
        I would be on the edge of my seat for all four hours of Stuxnet! The Musical. Sounded like a really fun class.
        danpalmer 3y ago
        Even better was the second year of the class (this lecturer lead the security side of the CS degrees). I took it in autumn 2013, just after the summer of Snowden. The class was just week after week of "I TOLD YOU SO! Look at these leaks! Look at these NSA details! Look at what we happens if we take this bit and stick it into LinkedIn and Google Street View and see where the developer of XKeyscore lives! Look at where this undersea cable goes, isn't that a bit close to Diego Garcia?! Look at this submarine and the bit carved out for an undersea cable splicing facility! I'VE BEEN SAYING THIS ALL FOR YEARS!" As much as it sounded like the rantings of a tin-foil hat wearing madman, he was in fact quite accurate through a combination of having worked in government in computer security at a fairly high level for a while, and having deep technical knowledge about what was possible. I miss his lectures, they were excellent, but I hear he's still going strong scaring freshers and running the on-campus teaching union, which is nice to hear.
        sitkack 3y ago
        I love playing BoE (back of envelope) Gedanken Experiments. Splicing a undersea cable or recording and analyzing all the SMS traffic in the world seems a little more tractable after you do the math.
  - analog31 3y ago
    What I noticed was the lack of motivation to study, and constant difficulty, eventually sapped the motivation to continue slogging through a degree and subsequent career just for the money. "I want to work in this field and make lots of money, but I hate it, and can't get through the coursework" is actually a pretty rare combination of talents. The path of being interested enough in a field that the easy work is easy, and the hard work is at least interesting, actually turns out to be the path of least resistance.
- fnordpiglet 3y ago
  I think LLMs are wonderful study aids and can bridge gaps in understanding by providing collaborative discovery with a “copilot,” but absolutely, if you cheat off of it, things catch up quickly. First, the sort of stuff ChatGPT can do out of the box is introductory course material. More advanced material requires a deep semantic understanding of the domain. Even if you use ChatGPT to produce boilerplate and syntax, or ask it for semantic advice, it won’t solve the crucial nuggets for you. Second, there is always a “test” component where you are required to remember what you learned impromptu without aids. Third, a lot of cs programs require collaborative work in projects. It’ll become very obvious who doesn’t know anything. Finally, even if you manage to graduate cheating through the entire program, when you interview for work, it’ll be transparent you learned nothing. A degree is an end, it’s a means to an end - be it research of industry, and experts will always be able to sniff out dilettantes.
alfor 3y ago
GPT is going to change everything, it’s not even a year in and most code is already written by AI. It save a lot of time but write a lot of local problem while ignoring the bigger picture. (for now) The only thing valuable is deep understanding of architecture and of the limitation of hardware. Know both end of the system, let the AI create all the glue in between.
- lawn 3y ago
  It's not even close to "most code". Most AI generated code isn't even correct and needs the human to give it some adjustments before it works as intended.
- lumost 3y ago
  I would be skeptical that "most" code is written by AI. Some code, certainly - but major enterprises are still terrified of the copyright outcome for existing non-local GPT implementations.
  - jazzyjackson 3y ago
    I noticed stable diffusion's CEO made a similarly fantastical claim, @13:33, "41 percent of all code on GitHub right now is AI generated" [0] Some redditors put together where the claim came from, it's a bad interpretation of Microsoft's Scott Guthrie, should be "40% of code checked in by users of github co-pilot" [1][2] [0] https://youtu.be/ciX_iFGyS0M https://youtu.be/ciX_iFGyS0M [1] https://www.reddit.com/r/github/comments/14qwhem/how_does_github_know_41_of_all_code_on_it_is_ai/ https://www.reddit.com/r/github/comments/14qwhem/how_does_gi... [2] https://www.microsoft.com/en-us/Investor/events/FY-2023/Morgan-Stanley-TMT-Conference https://www.microsoft.com/en-us/Investor/events/FY-2023/Morg...
    - lumost 3y ago
      I mean, I was a GitHub copilot user - but I haven’t checked in such code. I use gpt-4, but I only commit my experimental code.
- Swizec 3y ago
JimtheCoder 3y ago
By the time the kids are done their CS degree (4 years from now), are the LLMs going to be writing all the code anyways? If I were 18 again, I am not sure what the answer is - would it even be a good idea to pursue this field of study if there aren't going to be many jobs post graduation... I hope I am wrong...
- CatWChainsaw 3y ago
  Compare that to the whole genAI art thing and people here sneering that being an "Artist" is dead as a career. I wouldn't feel a lot of sympathy if such CSbros found themselves out of a job.
- nopinsight 3y ago
  I think humans need to move up the levels of abstraction and master specific domain knowledge. Barring AGI, there will still be need for humans to elicit and learn the requirements from customers and other stakeholders, then translate those into more concrete specifications. Future LFMs (Large Foundation Models, since they will be multimodal) can help automate some or much work from specifications onward. Humans will need to validate and revise the results by working with an LFM-powered system (Ref: the spiral model in software engineering).
  - chx 3y ago
    > then translate those into more concrete specifications. Yes and that's what code is: most concrete specifications :P Figuring out what to code is much harder than coding it, like 99.99% of the time. People worrying about ChatGPT taking away their job frankly do not understand this: my job is to think. I type very little code.
    - bcrosby95 3y ago
      Lots of new projects involve shitting out a massive amount of code though. I've certainly worked on stuff where I had precise clarity on what needed to be done and typing speed was practically the limiter. But for some other things it wasn't. I dunno. I go back and forth on the effects these things will have on programming.
TheOnly92 3y ago
I happen to be teaching a programming course currently, though it's not in English and the language I'm teaching is C. My current experience is that it does not seem like a majority of the students are using ChatGPT at all, even though I did encourage the use of it at the beginning of the course. For my own course, I think several factors contributed to students not utilizing ChatGPT as much: - The assignments are not in English, and performance of ChatGPT in languages other than English is subpar. - The programming language that I'm teaching is C, I'd imagine Python/Javascript and other more popular languages might lead to different outcomes - I did specifically design the assignments so that copy/pasting the assignment to ChatGPT does not lead to a usable answer (by restricting use of certain standard library functions, making the assignment more complicated) - The course is not introductory, i.e. a previous course already taught the basic syntax of C and basics of programming, so I can make my assignments much more advanced It's difficult to say if advancements in LLMs will make my job harder, where say copy/pasting my more complicated assignments can lead to correct results. But from what I can see right now, LLMs still have trouble solving novel problems, so it's probably always possible to come up with assignments that's difficult for them to solve.
- alach11 3y ago
  > from what I can see right now, LLMs still have trouble solving novel problems Have you been using 3.5 or 4?
  - TheOnly92 3y ago
    I have been using 3.5, but when I was designing the assignments, I asked a friend who had access to 4 to check it, and the code it produced was still incorrect.
- nsxwolf 3y ago
  I've had a few online code assessments that appear to have been hardened against ChatGPT "attacks". I failed at solving a problem that was just "Compute values of the Collatz conjecture for input n" because they wrote it to sound like an extremely difficult graph problem about being lost in a forest but being able to enter a "magic door". I fed the problem into ChatGPT later and it was utterly unable to comprehend it, but confidently gave wrong answer after wrong answer.
PaulDavisThe1st 3y ago
Drivel. A much more interesting consideration would be "Teaching Programming in the Age of Stack Overflow", which involves a well-established, gigantic resource for in-the-trenches programmers that generally provides functional, peer-reviewed and expansively commented solutions to the most common and sometimes most interesting problems. What programming will be like in an age where LLM's can correctly code is something we won't have any idea about for some time yet (if ever).
- VerminOctopus1 3y ago
  Huh? I have personally committed production code that was 100% generated by an LLM. Simply because the code was correct and written more quickly than I could have physically typed it out. Does that not count as correct code to you?
  - PaulDavisThe1st 3y ago
    Congratulations on being the exception that proves the rule, or rather, the minority situation that is being used to preemptively make claims about a technology that are not yet (and may never be) true. I tried it on my/our project 600K LOC (C++), all open source and almost certainly in the training set. Not only could it not explain the questions I asked it correctly, its answers and generated code were absolute jibberish, the sort of thing that would lead to you likely firing the person who gave them to you.
Barrin92 3y ago
I have no idea what the "age of chatgpt" is supposed to be, it changes nothing in regards to cheating. What my uni professor told us when I did my CS degree was. "You're not in kindergarten, if you want to copy your exercises you can copy them, but you have to do the exam at the end of the semester by hand without assistance and if you don't understand the exercises and do them yourself you're screwed". It's not like profs even bother to invent entirely new exercises every year so you could always just go through old course notes if you really wanted it easy. Doesn't matter if people Google, Stackoverflow, use ChatGpt, ask their wizkid neighbor or what have you. You don't need to resist cheaters, especially not when we're talking about adults, it's their responsibility. Cheaters don't learn so when they're tested they sooner or later flunk out.
hiAndrewQuinn 3y ago
I have the extremely fortunate scenario of watching my wife just begin a computer science degree now, with no prior experience programming whatsoever. This is not hyperbole: ChatGPT is an absolute game-changer when it comes to getting through the basics. It is simply far, far easier for a newbie to see a computer generate 1 / 3 / 5 slightly-different attempts to solve a specific problem they have, and then to pattern match sufficiently to be able to solve the problem themselves, than it is to muddle around by yourself with it for hours and hours with no end in sight. I thought she would have to ask me, someone who has been slinging Python in some form since 2008, a question at least once a week. In reality it's been about 3 times over the last 9 months. ChatGPT doesn't do much for experienced SWEs, but it demolishes the difficulty curve for newbies. I wish I had this when I was learning.
- gnicholas 3y ago
  > ChatGPT doesn't do much for experienced SWEs, but it demolishes the difficulty curve for newbies. Interesting, my impression was that ChatGPT helps experienced SWEs by filling out large amounts of boilerplate-ish code. It sounds like that's not your experience, though?
  - RussianCow 3y ago
    Which might make you, what, maybe 10% more productive overall? The more experienced you are, the less time you spend writing boilerplate code, so it's not that huge of a win in the grand scheme of things compared to what it provides when you're learning something new.
    - koonsolo 3y ago
      Not writing any unit tests are you?!?!? ;)
    - anonzzzies 3y ago
      It makes us far more productive; all of us, so we just do more as a team rather than fire someone. For me, copilot writing out if / then / else stuff like it's reading my mind, so tab -> 15 lines, all correct. Or mappings (.map .filter etc) which it completely does within milliseconds. As an experienced programmer, I cannot avoid these things and they do save me a lot of time and debugging. Oh yes, and tests. Going from a jira task to a bunch of jest tests for code that doesn't exist yet is a pretty big win. Very much more than 10% of the average day. You have to be sharp, be a good code reader and writer and it cannot do anything 'alone', but it makes us vastly more efficient.
nvy 3y ago
Perusing the comments here, it's interesting how short-sighted HN can be when it comes to LLMs. Generative "AI" is only going to get better, and the question of whether or not ChatGPT can write FizzBuzz, or avionics firmware, or a CRUD app, seems to be missing the forest for the trees. Even if LLMs or some successor technology never ends up supplanting programmers entirely, it is guaranteed we will see a mixture of deskilling (certain skills no longer being required, like spelling in the age of autocorrect) and massive productivity gains (meaning n-m workers can now do the job of n). These tools aren't going away, and they're only going to improve. Thus the market for programmers will shrink henceforth. There are no doubts about it. Maybe slowly, maybe quickly, but shrink it will, for the same reason that the market for radiologists is shrinking, or the reason that engineering firms no longer employ whole floors of draftsmen.
- ethanwillis 3y ago
  duck I guess I really should be worried.
- simion314 3y ago
  Assuming LLM improve productivity why do you think the market will shrink instead of getting even more stuff done? Say if some actually smart LLM could improve my team productivity with 25% then 1 we could fire a person 2 we could handle 25% more tasks, features and bugs So I think 2 will happen. My latest example of LLM issues,| I asked ChatGPT and competition to convert a line of jQuery into native JS, they all got it wrong because jQuery has some selectors like ":header" that is native to JS but the LLM used it anyway though somewhere in it's big memory it has the information that his is not native and if you prompt it right it will fix the issue. So seems to me the LLM are focusing too much on the prompt and failing to use it's full memory on the problem, so it can fix small individual micro tasks but is a complete waste of time for something a bit more advanced , my conclusion you can't have a manager or an artist armed with ChatGPT and create a full project without actually learning to code, at best they will learn to code from the LLM but the hard way and probably using outdated code and inefficient ways.
  - nvy 3y ago
Afzaly 3y ago
DSl)program-teacher52344g
deleted 3y ago
[deleted]
kesor 3y ago
Not just programming. If you are interested in any kind of topic, be it scientific, or humanistic, GPT is an amazing tool to slash that learning curve. Just as an example, try using ChatGPT to explain an article in a foreign language. Can even go as low as a letter-by-letter break down of each word. No private tutor will ever have the patience to teach people at this level.
bawana 3y ago
Not using chatGPT to code is like not using antibiotics to treat infection. Sure, it can be done but why go back to the old ways? How many people know how to treat infections without antibiotics? How many people know how the antibiotics they use actually work ? Coding will become like the liturgical exercises of monks toiling in isolated monasteries while the rest of the world will move on. I cant wait for a chatGPT decompiler. IDA, watch out!
- Tyr42 3y ago
  That analogy does open you up to the natural comparison of overuse of antibiotics leading to resistant superbugs doesn't it though. Hopefully our bugs aren't evolving though.
  - __loam 3y ago
    That comparison is why this is a bad analogy.
    - HappMacDonald 3y ago
      .. that over-reliance on things one does not understand guarantees one will get stuck in new problems said tool can no longer solve?
      - __loam 3y ago
        Point taken
keenmaster 3y ago
>CTRL+F GPT4, GPT 4, GPT-4 Disappointing. I cannot evaluate claims about GPT when people say ChatGPT instead of GPT4, if they actually mean the latter. And if someone really is discussing the usefulness of AI while only having used ChatGPT (GPT3.5), then they’re missing out on a major improvement and their input is less valuable than those discussing GPT 4. Of course GPT-4 has plenty of limitations but people just need to be clear that they’re familiar with the state of the art. Also, statements to the effect of “it just predicts the next word” do not appreciate the major difference in capacity for learned abstractions between GPT 3.5 vs GPT 4. So to me it’s just not a useful way of thinking about LLMs. It may technically be true, but at some level that can also be said of human beings. In other news, an airplane “just” flies.
pmulard 3y ago
I see ChatGPT as the next “layer” on top of google search in the greater scheme of information. Before google, one had to spend a lot of time perusing the internet manually. And before that, people had to read physical books or call someone up and have a conversation over the phone. Google drastically sped up that process by giving you the information in 2 seconds, for what would normally take a few hours. ChatGPT is a game changer for this same process. It speeds up the googling that would normally take a few hours and gives me my information in a few queries. There have been so many times where I’m reading a book and need to research a bunch of technical terms it throws at me; which then consists of wading through blog posts or documentation, with varying levels of difficulty and quality. This process is sped up 100x because of chat GPT, because I can followup with questions and customize them to my specific application. Sometimes I need things explained like I’m five; others, a deep technical deep dive. Point is, it allows for dynamic interaction with content; it helps when I’m struggling with a bug, documentation, a book I’m reading, or pieces of code myself or someone else has written. It’s been an absolute game changer.

154 points by headalgorithm 3y ago

I knew a few people who thought they could cheat their way through a CS degree. They all dropped out or changed majors within a few semesters. It’s generally a problem that solves itself, in my experience. Perhaps a benefit of these tools is that we stop the obsession over cheating, which inconveniences honest students in many ways. Cheating has always occurred, but now we can’t even pretend that it’s preventable.

NationOfJoe 3y ago
Would you be able to elaborate on why they dropped out? did they hit a point where cheating wasn't possible? or maybe realized it was pointless and unhelpful once they need to do a job?
- paxys 3y ago
  In my experience it's always the former. Back in university the surest sign of a good class was that you were allowed to "cheat". The professor encouraged using any and all outside materials for your assignments. You were allowed to work with other students on projects. Exams were all open book. You could use open source code. Such classes always led to the best outcomes for students who were truly interested and engaged, and those who weren't had no way of faking it. On the other hand classes where professors spent more effort on detecting cheating than teaching the material were always duds.
  - danpalmer 3y ago
    Huge +1 to this. I had one lecturer who would essentially rant at us about security vulnerabilities every week for a semester, and then write a paper based on his rants. Some students ignored his rants because they didn't find the subject interesting, did the past papers, and then got to the exam and did terribly, then complained about the fact that "it's all based on his rants". Others found the ranting engaging because it was deep dives on obscure bits of computer security history. My year he spent ~4 weeks going on about Stuxnet, including deep dives into the wider political context. When I got to the paper, one of the few questions we could choose, for 50% of the paper, was just "What was Stuxnet". I wrote pages and pages. Figured out from marking that I got full marks on that one. I always did great in that lecturer's modules despite never taking notes and rarely doing any targeted reading. To show the level of engagement, I once turned up to one of these lectures 10 mins late, with most of the class already there, and the lecturer said "oh I wondered where you guys were, I guess I'll start again". He knew we were the only ones who cared about the class.
    - sitkack 3y ago
      I would be on the edge of my seat for all four hours of Stuxnet! The Musical. Sounded like a really fun class.
      - danpalmer 3y ago
        Even better was the second year of the class (this lecturer lead the security side of the CS degrees). I took it in autumn 2013, just after the summer of Snowden. The class was just week after week of "I TOLD YOU SO! Look at these leaks! Look at these NSA details! Look at what we happens if we take this bit and stick it into LinkedIn and Google Street View and see where the developer of XKeyscore lives! Look at where this undersea cable goes, isn't that a bit close to Diego Garcia?! Look at this submarine and the bit carved out for an undersea cable splicing facility! I'VE BEEN SAYING THIS ALL FOR YEARS!" As much as it sounded like the rantings of a tin-foil hat wearing madman, he was in fact quite accurate through a combination of having worked in government in computer security at a fairly high level for a while, and having deep technical knowledge about what was possible. I miss his lectures, they were excellent, but I hear he's still going strong scaring freshers and running the on-campus teaching union, which is nice to hear.
        sitkack 3y ago
        I love playing BoE (back of envelope) Gedanken Experiments. Splicing a undersea cable or recording and analyzing all the SMS traffic in the world seems a little more tractable after you do the math.
- analog31 3y ago
  What I noticed was the lack of motivation to study, and constant difficulty, eventually sapped the motivation to continue slogging through a degree and subsequent career just for the money. "I want to work in this field and make lots of money, but I hate it, and can't get through the coursework" is actually a pretty rare combination of talents. The path of being interested enough in a field that the easy work is easy, and the hard work is at least interesting, actually turns out to be the path of least resistance.
fnordpiglet 3y ago
I think LLMs are wonderful study aids and can bridge gaps in understanding by providing collaborative discovery with a “copilot,” but absolutely, if you cheat off of it, things catch up quickly. First, the sort of stuff ChatGPT can do out of the box is introductory course material. More advanced material requires a deep semantic understanding of the domain. Even if you use ChatGPT to produce boilerplate and syntax, or ask it for semantic advice, it won’t solve the crucial nuggets for you. Second, there is always a “test” component where you are required to remember what you learned impromptu without aids. Third, a lot of cs programs require collaborative work in projects. It’ll become very obvious who doesn’t know anything. Finally, even if you manage to graduate cheating through the entire program, when you interview for work, it’ll be transparent you learned nothing. A degree is an end, it’s a means to an end - be it research of industry, and experts will always be able to sniff out dilettantes.

alfor 3y ago

GPT is going to change everything, it’s not even a year in and most code is already written by AI. It save a lot of time but write a lot of local problem while ignoring the bigger picture. (for now) The only thing valuable is deep understanding of architecture and of the limitation of hardware. Know both end of the system, let the AI create all the glue in between.

lawn 3y ago
It's not even close to "most code". Most AI generated code isn't even correct and needs the human to give it some adjustments before it works as intended.
lumost 3y ago
I would be skeptical that "most" code is written by AI. Some code, certainly - but major enterprises are still terrified of the copyright outcome for existing non-local GPT implementations.
- jazzyjackson 3y ago
  I noticed stable diffusion's CEO made a similarly fantastical claim, @13:33, "41 percent of all code on GitHub right now is AI generated" [0] Some redditors put together where the claim came from, it's a bad interpretation of Microsoft's Scott Guthrie, should be "40% of code checked in by users of github co-pilot" [1][2] [0] https://youtu.be/ciX_iFGyS0M https://youtu.be/ciX_iFGyS0M [1] https://www.reddit.com/r/github/comments/14qwhem/how_does_github_know_41_of_all_code_on_it_is_ai/ https://www.reddit.com/r/github/comments/14qwhem/how_does_gi... [2] https://www.microsoft.com/en-us/Investor/events/FY-2023/Morgan-Stanley-TMT-Conference https://www.microsoft.com/en-us/Investor/events/FY-2023/Morg...
  - lumost 3y ago
    I mean, I was a GitHub copilot user - but I haven’t checked in such code. I use gpt-4, but I only commit my experimental code.
Swizec 3y ago

JimtheCoder 3y ago

By the time the kids are done their CS degree (4 years from now), are the LLMs going to be writing all the code anyways? If I were 18 again, I am not sure what the answer is - would it even be a good idea to pursue this field of study if there aren't going to be many jobs post graduation... I hope I am wrong...

CatWChainsaw 3y ago
Compare that to the whole genAI art thing and people here sneering that being an "Artist" is dead as a career. I wouldn't feel a lot of sympathy if such CSbros found themselves out of a job.
nopinsight 3y ago
I think humans need to move up the levels of abstraction and master specific domain knowledge. Barring AGI, there will still be need for humans to elicit and learn the requirements from customers and other stakeholders, then translate those into more concrete specifications. Future LFMs (Large Foundation Models, since they will be multimodal) can help automate some or much work from specifications onward. Humans will need to validate and revise the results by working with an LFM-powered system (Ref: the spiral model in software engineering).
- chx 3y ago
  > then translate those into more concrete specifications. Yes and that's what code is: most concrete specifications :P Figuring out what to code is much harder than coding it, like 99.99% of the time. People worrying about ChatGPT taking away their job frankly do not understand this: my job is to think. I type very little code.
  - bcrosby95 3y ago
    Lots of new projects involve shitting out a massive amount of code though. I've certainly worked on stuff where I had precise clarity on what needed to be done and typing speed was practically the limiter. But for some other things it wasn't. I dunno. I go back and forth on the effects these things will have on programming.

TheOnly92 3y ago

I happen to be teaching a programming course currently, though it's not in English and the language I'm teaching is C. My current experience is that it does not seem like a majority of the students are using ChatGPT at all, even though I did encourage the use of it at the beginning of the course. For my own course, I think several factors contributed to students not utilizing ChatGPT as much: - The assignments are not in English, and performance of ChatGPT in languages other than English is subpar. - The programming language that I'm teaching is C, I'd imagine Python/Javascript and other more popular languages might lead to different outcomes - I did specifically design the assignments so that copy/pasting the assignment to ChatGPT does not lead to a usable answer (by restricting use of certain standard library functions, making the assignment more complicated) - The course is not introductory, i.e. a previous course already taught the basic syntax of C and basics of programming, so I can make my assignments much more advanced It's difficult to say if advancements in LLMs will make my job harder, where say copy/pasting my more complicated assignments can lead to correct results. But from what I can see right now, LLMs still have trouble solving novel problems, so it's probably always possible to come up with assignments that's difficult for them to solve.

alach11 3y ago
> from what I can see right now, LLMs still have trouble solving novel problems Have you been using 3.5 or 4?
- TheOnly92 3y ago
  I have been using 3.5, but when I was designing the assignments, I asked a friend who had access to 4 to check it, and the code it produced was still incorrect.
nsxwolf 3y ago
I've had a few online code assessments that appear to have been hardened against ChatGPT "attacks". I failed at solving a problem that was just "Compute values of the Collatz conjecture for input n" because they wrote it to sound like an extremely difficult graph problem about being lost in a forest but being able to enter a "magic door". I fed the problem into ChatGPT later and it was utterly unable to comprehend it, but confidently gave wrong answer after wrong answer.

PaulDavisThe1st 3y ago

Drivel. A much more interesting consideration would be "Teaching Programming in the Age of Stack Overflow", which involves a well-established, gigantic resource for in-the-trenches programmers that generally provides functional, peer-reviewed and expansively commented solutions to the most common and sometimes most interesting problems. What programming will be like in an age where LLM's can correctly code is something we won't have any idea about for some time yet (if ever).

VerminOctopus1 3y ago
Huh? I have personally committed production code that was 100% generated by an LLM. Simply because the code was correct and written more quickly than I could have physically typed it out. Does that not count as correct code to you?
- PaulDavisThe1st 3y ago
  Congratulations on being the exception that proves the rule, or rather, the minority situation that is being used to preemptively make claims about a technology that are not yet (and may never be) true. I tried it on my/our project 600K LOC (C++), all open source and almost certainly in the training set. Not only could it not explain the questions I asked it correctly, its answers and generated code were absolute jibberish, the sort of thing that would lead to you likely firing the person who gave them to you.

Barrin92 3y ago

I have no idea what the "age of chatgpt" is supposed to be, it changes nothing in regards to cheating. What my uni professor told us when I did my CS degree was. "You're not in kindergarten, if you want to copy your exercises you can copy them, but you have to do the exam at the end of the semester by hand without assistance and if you don't understand the exercises and do them yourself you're screwed". It's not like profs even bother to invent entirely new exercises every year so you could always just go through old course notes if you really wanted it easy. Doesn't matter if people Google, Stackoverflow, use ChatGpt, ask their wizkid neighbor or what have you. You don't need to resist cheaters, especially not when we're talking about adults, it's their responsibility. Cheaters don't learn so when they're tested they sooner or later flunk out.

hiAndrewQuinn 3y ago

I have the extremely fortunate scenario of watching my wife just begin a computer science degree now, with no prior experience programming whatsoever. This is not hyperbole: ChatGPT is an absolute game-changer when it comes to getting through the basics. It is simply far, far easier for a newbie to see a computer generate 1 / 3 / 5 slightly-different attempts to solve a specific problem they have, and then to pattern match sufficiently to be able to solve the problem themselves, than it is to muddle around by yourself with it for hours and hours with no end in sight. I thought she would have to ask me, someone who has been slinging Python in some form since 2008, a question at least once a week. In reality it's been about 3 times over the last 9 months. ChatGPT doesn't do much for experienced SWEs, but it demolishes the difficulty curve for newbies. I wish I had this when I was learning.

gnicholas 3y ago
> ChatGPT doesn't do much for experienced SWEs, but it demolishes the difficulty curve for newbies. Interesting, my impression was that ChatGPT helps experienced SWEs by filling out large amounts of boilerplate-ish code. It sounds like that's not your experience, though?
- RussianCow 3y ago
  Which might make you, what, maybe 10% more productive overall? The more experienced you are, the less time you spend writing boilerplate code, so it's not that huge of a win in the grand scheme of things compared to what it provides when you're learning something new.
  - koonsolo 3y ago
    Not writing any unit tests are you?!?!? ;)
  - anonzzzies 3y ago
    It makes us far more productive; all of us, so we just do more as a team rather than fire someone. For me, copilot writing out if / then / else stuff like it's reading my mind, so tab -> 15 lines, all correct. Or mappings (.map .filter etc) which it completely does within milliseconds. As an experienced programmer, I cannot avoid these things and they do save me a lot of time and debugging. Oh yes, and tests. Going from a jira task to a bunch of jest tests for code that doesn't exist yet is a pretty big win. Very much more than 10% of the average day. You have to be sharp, be a good code reader and writer and it cannot do anything 'alone', but it makes us vastly more efficient.

nvy 3y ago

Perusing the comments here, it's interesting how short-sighted HN can be when it comes to LLMs. Generative "AI" is only going to get better, and the question of whether or not ChatGPT can write FizzBuzz, or avionics firmware, or a CRUD app, seems to be missing the forest for the trees. Even if LLMs or some successor technology never ends up supplanting programmers entirely, it is guaranteed we will see a mixture of deskilling (certain skills no longer being required, like spelling in the age of autocorrect) and massive productivity gains (meaning n-m workers can now do the job of n). These tools aren't going away, and they're only going to improve. Thus the market for programmers will shrink henceforth. There are no doubts about it. Maybe slowly, maybe quickly, but shrink it will, for the same reason that the market for radiologists is shrinking, or the reason that engineering firms no longer employ whole floors of draftsmen.

ethanwillis 3y ago
duck I guess I really should be worried.
simion314 3y ago
Assuming LLM improve productivity why do you think the market will shrink instead of getting even more stuff done? Say if some actually smart LLM could improve my team productivity with 25% then 1 we could fire a person 2 we could handle 25% more tasks, features and bugs So I think 2 will happen. My latest example of LLM issues,| I asked ChatGPT and competition to convert a line of jQuery into native JS, they all got it wrong because jQuery has some selectors like ":header" that is native to JS but the LLM used it anyway though somewhere in it's big memory it has the information that his is not native and if you prompt it right it will fix the issue. So seems to me the LLM are focusing too much on the prompt and failing to use it's full memory on the problem, so it can fix small individual micro tasks but is a complete waste of time for something a bit more advanced , my conclusion you can't have a manager or an artist armed with ChatGPT and create a full project without actually learning to code, at best they will learn to code from the LLM but the hard way and probably using outdated code and inefficient ways.
- nvy 3y ago

Afzaly 3y ago

DSl)program-teacher52344g

deleted 3y ago

[deleted]

kesor 3y ago

Not just programming. If you are interested in any kind of topic, be it scientific, or humanistic, GPT is an amazing tool to slash that learning curve. Just as an example, try using ChatGPT to explain an article in a foreign language. Can even go as low as a letter-by-letter break down of each word. No private tutor will ever have the patience to teach people at this level.

bawana 3y ago

Not using chatGPT to code is like not using antibiotics to treat infection. Sure, it can be done but why go back to the old ways? How many people know how to treat infections without antibiotics? How many people know how the antibiotics they use actually work ? Coding will become like the liturgical exercises of monks toiling in isolated monasteries while the rest of the world will move on. I cant wait for a chatGPT decompiler. IDA, watch out!

Tyr42 3y ago
That analogy does open you up to the natural comparison of overuse of antibiotics leading to resistant superbugs doesn't it though. Hopefully our bugs aren't evolving though.
- __loam 3y ago
  That comparison is why this is a bad analogy.
  - HappMacDonald 3y ago
    .. that over-reliance on things one does not understand guarantees one will get stuck in new problems said tool can no longer solve?
    - __loam 3y ago
      Point taken

keenmaster 3y ago

>CTRL+F GPT4, GPT 4, GPT-4 Disappointing. I cannot evaluate claims about GPT when people say ChatGPT instead of GPT4, if they actually mean the latter. And if someone really is discussing the usefulness of AI while only having used ChatGPT (GPT3.5), then they’re missing out on a major improvement and their input is less valuable than those discussing GPT 4. Of course GPT-4 has plenty of limitations but people just need to be clear that they’re familiar with the state of the art. Also, statements to the effect of “it just predicts the next word” do not appreciate the major difference in capacity for learned abstractions between GPT 3.5 vs GPT 4. So to me it’s just not a useful way of thinking about LLMs. It may technically be true, but at some level that can also be said of human beings. In other news, an airplane “just” flies.

pmulard 3y ago

I see ChatGPT as the next “layer” on top of google search in the greater scheme of information. Before google, one had to spend a lot of time perusing the internet manually. And before that, people had to read physical books or call someone up and have a conversation over the phone. Google drastically sped up that process by giving you the information in 2 seconds, for what would normally take a few hours. ChatGPT is a game changer for this same process. It speeds up the googling that would normally take a few hours and gives me my information in a few queries. There have been so many times where I’m reading a book and need to research a bunch of technical terms it throws at me; which then consists of wading through blog posts or documentation, with varying levels of difficulty and quality. This process is sped up 100x because of chat GPT, because I can followup with questions and customize them to my specific application. Sometimes I need things explained like I’m five; others, a deep technical deep dive. Point is, it allows for dynamic interaction with content; it helps when I’m struggling with a bug, documentation, a book I’m reading, or pieces of code myself or someone else has written. It’s been an absolute game changer.