5 ms·

Coding on Copilot: Data suggests downward pressure on code quality

• PreInternet01 3y ago
  Discussed previously: https://news.ycombinator.com/item?id=39168105 https://news.ycombinator.com/item?id=39168105
• datadrivenangel 3y ago
  "GitClear analyzed approximately 153 million changed lines of code, authored between January 2020 and December 2023 [A1]. This is the largest known database of highly structured code change data that has been used to evaluate code quality differences [A2]. We find disconcerting trends for maintainability. Code churn -- the percentage of lines that are reverted or updated less than two weeks after being authored -- is projected to double in 2024 compared to its 2021, pre-AI baseline. We further find that the percentage of "added code" and "copy/pasted code" is increasing in proportion to “updated,” “deleted,” and “moved” code. In this regard, code generated during 2023 more resembles an itinerant contributor, prone to violate the DRY-ness of the repos visited. " If the cost of updating code has dropped due to AI, is code churn still a good proxy for low code quality?
  • tigeroil 3y ago
    Indeed - if anything in my organisation AI has lead to developers replacing poor quality existing code more quickly because what was quite a time consuming process before can be sped up tremendously by using language models.
    • dartos 3y ago
      Replacing poor code, with poor code that nobody really understands isn’t good either. Imagine hiring a junior to write your code, except they’d just stop existing when they were done. You’d need to ask a whole new junior to explain it to you later (if you were relying on AI) At the recent startup I was at, the CEO (who was also an engineer) basically relied on chatgpt to make the whole app. It was a complicated and fragile codebase. The docstrings, which were also AI generated, were frequently out of date if not just wrong. We were in the midst of rewriting it before we went bust.
      • lucioperca 3y ago
        Well if you need it touch it again, you hope for an even better AI.
        • dartos 3y ago
          lol, I mean I guess that’s one way to do it.
      • hedora 3y ago
        Their metric doesn’t include old poor quality code that was recently replaced. It only looks for brand new code that was immediately reverted/rewritten, and also looks for code that was copied from the internet vs. from within the repo (the former is probably non-idiomatic for the code base in question).
  • lostmsu 3y ago
    This could also be an indication of young codebases or more frequent commits. Basically, the AI correlation is baseless.
• naasking 3y ago
  Possible downward pressure on code quality, but maybe it's partly because more people are getting involved in programming with coding assistance? That could be ok if there's more progress overall.
  • tussa 3y ago
    Would it? What does 1% reduction in code quality translate to in terms of delivery time and profits? I don't know. But I'm pretty sure 50% reduction results in project failure and company bankruptcy.
    • lkbm 3y ago
      Depends where we're seeing this. I worked as a high school bureaucrat for several years, and through some simple scripts, I was able to make some tedious data entry tasks vastly faster. It was ugly, hacked together code, with lots of hardcoded values I had to update each semester, but it worked and was way better than doing it all by hand. Low quality code is worse than good quality code, but often better than manual labor. I want my web browser and my bank to use high quality code, but I want semi-technical people who would otherwise do everything manually to be able to automate tedious portions of their jobs. (The caveat, of course, being that a buggy script can screw up the thing the thing it's automating must faster than a manual process as well.)
    • hedora 3y ago
      The study shows a doubling in the rate at which tech debt code is produced and checked into the repo. Anecdotally, as a principal engineer, I’ve definitely noticed that new senior engineers on the team that say they are using chatgpt/copilot produce unprecedentedly bad code at unprecedented rates. It takes me 2-3x longer to unwind such crap than it would for me to write it from scratch. As we grow the team, this will definitely put us out of business unless we find a way to fix it. Currently, we’re hoping the AI assisted engineers will get better at unborking code before merging it, but that’s a harder task than RTFM or going to stack overflow to copy-paste.
      • 3y ago
• hackernoteng 3y ago
  Gee really? You mean generated code by some ML model means people get lazier and just cut/paste hallucinated code? Who would have thought?
  • gtirloni 3y ago
    I doubt AI is making anyone lazier and less worried about code quality, if they weren't already.
    • cjbgkagh 3y ago
      People can learn things even if they would rather not. Allowing them to take shortcuts will diminish their learning opportunities.
      • flir 3y ago
        I'm not ashamed to admit I'm learning a lot from the chatbots. And it seems to stick better, as compared to reading a manual.
        • gtirloni 3y ago
          Same here. Instead of me conforming my thoughts to some document, the chatbot works together with me and my questions. I find them very useful form learning, so much I usually start a learning session first before I ever try to search something on Google. Back to code solutions, I never ever use them if I don't understand ALL that is happening in that code. I know the bot will ignore many corner cases or that it doesn't know some subtle implications about the context where I'll deploy the code. I think someone that doesn't care about code quality was already copying and pasting code from StackOverflow and fisting the keyboard until it works then moving to the next ticket. I really don't think AI is making things worse in that regard.
    • fhd2 3y ago
      Maybe lazy is the wrong term, but less attentive apparently. There was the Ironies of Automation paper about this: https://ckrybus.com/static/papers/Bainbridge_1983_Automatica.pdf https://ckrybus.com/static/papers/Bainbridge_1983_Automatica...
• ogogmad 3y ago
  Are there "best practices" that can help here?
  • orangesite 3y ago
    Yeah, don't hire a parrot to do an engineer's job.
    • ogogmad 3y ago
      That's sneery bullshit.
• evilantnie 3y ago
  Copilot is a lossy compression algorithm when applied at scale, I would expect some degradation in code quality if not applied appropriately. It's still a useful tool, but just like image and audio compression, a human needs to give the final output some last looks to ensure an acceptable out come.
• nicup12345689 3y ago
  plus ça change, plus c'est la même chose
• stcredzero 3y ago
  I think this is what's happening. Code production speed has increased. But the speed and efficiency of review and testing hasn't kept pace. Also, my current job limits the use of AI coding tools, because of the risk of our IP leaking out. To really be useful, there is a lot of context a programmer should be able to collate. Not all of it is going to be in the repo.
  • dathinab 3y ago
    IMHO there is some form of crisis of "value production speed" for various programming things. And AI gives you fast code production speed so it seems to be a fix. But due to less structured code and code reuse it needs much more code for the same value produced. And it loves to produce the kind of bugs which are easy to pass reviews and tests (because this are the kind of bugs it most likely has as "correct" code in it's training data). Which means less time per-value for human code reviews and in turn lower quality results on many levels. I originally hoped that with CS/SoftwareDev getting older and less explosive grows and Universities more realizing the (sometimes huge) gap between CS and SoftwareDev the industry would get a chance of consolidating tech and dev processes to fix that issue. But with the rise of LLMs this hope might be further away then it ever had been in the last 20 year.
    • stcredzero 3y ago
      Why not devote LLMs to enhancing testing and review?
      • dathinab 3y ago
        the current code generating LLMs seem not suited for this just having more test doesn't mean you actually test more (or more correctly) Having AI specialized on assisting testing is viable but in the end would be done quite different as far as I can tell even if it uses LLMs. Through maybe domain adoption and a very different scaffolding might be enough.
        • stcredzero 3y ago
• ape4 3y ago
  Just wait until this AI-generated code is posted on github and is read as the source for the next version of Copilot.
• 1B05H1N 3y ago
  I've found that it's useful for explaining things to me, and barely anything else.
  • chaxor 3y ago
    It's ok if you: 1) Tell it what you want 2) Then realize it messed up because you forgot to specify more details 3) Repeat 1) & 2) ~25-50 (maybe>100) times, with different restarts/contexts with a clear goal of style and performance in mind. 4) Make some minor adjustments on your own Now you have that 6 line script you wanted. Honestly it is still pretty useful though, as you can see 'your code in different clothing styles' in some sense, very quickly to choose the best/most performant option. You just have to know what that looks like before going in, and it helps make it 'tangible' quicker.
• jfisher4024 3y ago
  I used to use it mainly to write unit tests, which is a nice use case because it’s obvious if it gets it wrong. (The test fails.) One time I had to use the Go protobuff reflection API, and Copilot was actual crucial because the docs are so bad. Can’t use it in my current role and I feel less productive.
  • pinkmuffinere 3y ago
    This seems to me like the perfect use case, I’ll have to try using it for unit tests. Are you ever concerned when the AI-written unit test passes that it might be passing incorrectly? Either for the wrong reason, or when it should fail?
• deleted 3y ago
  [deleted]
• deleted 3y ago
  [deleted]
• from-nibly 3y ago
  Code reuse is not a virtue it's a tool. WET is a good way to write code anyway. And at the end of the day, programmers who don't know what they are really trying to accomplish are never going to write good code. AI will just help them write different bad code faster.
• dawitbelai408 3y ago
  [dead]
• nzach 3y ago
  >[...]"Churn" is the percentage of code that was pushed to the repo, then subsequently reverted, removed or updated within 2 weeks. This was a relatively infrequent outcome when developers authored all their own code -- only 3-4% of code was churned prior to 2023, although there is also a hint of the coming uptick in 2022, when Churn jumped 9%. 2022 was the first year Copilot was available in beta, and the year that ChatGPT became available. I think this is a bad take on this issue. It implies that if either chatgpt or the devs were just "smarter" this would not happen. This also implies that churn is a bad thing. And I don't think neither of these are true. The real problem is that debugging is twice as hard as programming and now for the first time we got to the point that people can create software that "works" but they cannot fully grasp why exactly it works. Of course you can go with the easy answer (these people are just lazy/dumb and should not be in this field) but this doesn't help. But the reality is that we don't really have anything better than printf and gdb to understand what the is happening inside our computers. Hopefully now we will have more incentives to create better tools to help our understanding of what happens under the hood.
  • squigz 3y ago
    > we got to the point that people can create software that "works" but they cannot fully grasp why exactly it works. It seems to me that this is the real problem. The difference between a sub-par programmer and a competent one is that the latter can create code that they understand and can therefore more easily debug. Being able to create code that "works", but you then have to spend hours/days trying to grok in order to debug it, seems to me to be more harmful than productive.
  • stinos 3y ago
    for the first time we got to the point that people can create software that "works" but they cannot fully grasp why exactly it works. The huge amount of times I saw //This makes everything work somehow sleep(1) begs to differ