22 ms·
New GitHub Copilot research finds 'downward pressure on code quality'
- morphicpro 3y ago[flagged]
- ametrau 3y agoEventually meal mixes many ingredients, low quality, and gives someone diarrhoea and no one can figure out why.
- morphicpro 3y ago[flagged]
- shermantanktop 3y agoChef would rather tell line cook to make mayonnaise from scratch even though there is a giant tub on the next table.
- deleted 3y ago[deleted]
- ok123456 3y agoCode churn is up because the cost of trying new ideas is cheaper. Not all of the churn is because the code was defective.
- dangus 3y agoThe amount of things I’m expected to be able to accomplish constantly increases. I get it, I’m becoming more senior, but at the same time, it’s kind of crazy that a single person can just be told to do the kind of things I’m told to do with the kind of deadlines I have. I can think of some takes I do by myself with code where a decade or two ago I’d be talking about hiring dozens of employees or contractors across the globe to physically enter a building and make a global business information system possible. I think this is related to the productivity wage gap. I’m being paid lower than I was in 2018 adjusted for inflation and I’m far more capable, producing far more value for my employer. As my tools get more valuable my labor gets proportionally less valuable. (A certain historical figure predicted this end state of capitalism). Here’s another take: I don’t give a fuck about the code quality at my company. I’m not paid extra to write good code. My bonus is based on basically nothing I can influence. If the whole company goes out of business on Monday I’ll enjoy my long weekend.
- SoftTalker 3y agoThis is how everything goes. To build a house, it used to be necessary to hire experienced craftsmen. Now most of it is delivered pre-fabricated to the building site and laborers hired for the day nail it together. Cars used to be hand built by machinists and metalworkers. Now laborers tighten a bolt as cars roll by on an assembly line. Or a robot does it. Computer coding is going the same way. We lived and worked through the craftsman stage. Now we're becoming laborers.
- extheat 3y agoThis is why I am concerned that really good AI tools (be it LLMs, or not), are only going to make people dumber over time. It is true that those people that are highly motivated will be digitally enhanced by LLMs, but the hard reality is that this does not encompass most people. Most people will do the least amount of work possible to solve a task. Rather than taking the time to understand and critically reason about the output of an LLM, these people will instead take it and run, only asking questions later [if at all]. And as we zoom out and think more farther into the future, I see it getting much worse. If AI is really doing all the "hard stuff", then the general human incentive to learn and do them at all quickly treads to 0. This isn't going to be everyone, I think some people will absolutely become "10x developers" or the equivalent for other domains. But all this will result in is more inequity, in my naïve view. The universe has a fundamental property that things move from higher energy states into lower ones. From a human POV, I think you could apply a similar idea, if the need to be smart quickly recedes, then in general we may degrade unaugmented human collective intelligence over time. I don't know, maybe we'll figure something out to make it much easier to learn and ingest new concepts, but it seems more and more to me that the high obstacles for human brains learning things (with poor memory) is too big a bottleneck to overcome any time soon.
- badrequest 3y ago"This tech will only make people dumber over time" was also claimed about the Internet.
- IlliOnato 3y ago"Internet makes smart people smarter and dumb people dumber". (Mark Solonin, a Russian historian). Taking into account that such blunt statements always hide a lot of nuances, this seems to capture the reality.
- manvillej 3y agoyes but in this case, we have actual research showing that the technology is degrading the quality of work.
- mrweasel 3y agoPeople have different workflows, but mine is frequently, skim the documentation, make a prototype, refine code a bit, add tests, move stuff around, break stuff, rework code, study documentation, refactor a bit more, and then at that point I have enough understanding of the problem to go in at yank out 80% of my code and do it right. If Copilot gives me working code in the prototype stage, good enough that I can just move on to the next thing, my understanding is never going to be good enough that I can go in and structure everything correctly. It will effectively allow me to skip 90% of my workflow, but pay the price. That's not to say that Copilot can't be extremely helpful during the final steps of development. If those findings are correct, I can't say that I'm surprised. Bad code is written by poor understanding and Copilot can't have any understanding beyond what you provide it. It may write better code than the average programmer, but the result is no better than the input given. People are extremely focused on "prompt engineering", so why act surprised when a poor "prompt" in VScode yields a poor result?
- andybak 3y agoI'm not sure why you decided that "use copilot" also implies missing out most of your later steps. Who decides to skip all those steps? Presumably you? My experience is that Copilot is great at getting me started. Sometimes the code is good, sometimes it's mediocre or completely broken. But it's invaluable at getting me thinking. I wasted a lot more time before I started using it. That might just be my weird brain wiring... (Edited to sound less narky. I shouldn't post from a mobile device)
- fwsgonzo 3y agoI recently tried Copilot out of curiosity and this is my experience too: It helps me getting started, which for me is 99% of the challenge. I know how to solve problems, even complex ones, but for some reason getting started is just so extremely hard, sometimes. Copilot lets me get started, even if it's wrong sometimes. There have been times where I have been surprised by how it took something I wrote for a server, and presented the correct client-side implementation. I've used it a few times to describe a problem and let it handle the solution. It's not very good, but I wonder if one should place more blame on PEBCAK and put more time into problem-description. I gave it a few more paragraphs to describe the problem, and eventually I could take it from there. It was still wrong, but enough to get me started. Immensely helpful that way. Another aspect that I'm wondering about is if it will be able to do more with better documented code. Anyone have experience with that? I've started to write more doxygen comments, and hoping to see if there's a slow shift to more accurate predictions.
- simonw 3y agoThe full paper is here: https://gitclear-public.s3.us-west-2.amazonaws.com/Coding-on-Copilot-2024-Developer-Research.pdf https://gitclear-public.s3.us-west-2.amazonaws.com/Coding-on...
- stusmall 3y agoThank you for the source.
- padjo 3y agoThe real news here is that the authors have apparently found an objective and universal measure of code quality.
- o11c 3y agoEh, it's not that such measures don't exist - they're just noisy. And the thing about AI is that their negative impact is clearly visible above the noise.
- godzillabrennus 3y agoI decided to use ChatGPT to build a clone of Yourls using Django/Python. I gave it specific instructions to not only allow for a custom shortened URL but to track the traffic. It didn’t properly contemplate how to do that in the logic or data model. I had to feed it specific instructions afterwards to get it fixed. AI tools are akin to having a junior developer working for you. Except they are much much faster. If you don’t know what you’re doing they just accelerate the pace that you make mistakes.
- KronisLV 3y ago> AI tools are akin to having a junior developer working for you. Except they are much much faster. Honestly, this is brilliant. The other day I had to add table name prefixes to a SELECT statement column aliases, since such a feature just doesn't exist for some reason, a bit like: -- fails because of duplicate column names (e.g. when creating a view) SELECT * FROM table_a JOIN table_b ON ... JOIN table_c ON ... ... -- this would solve my issue, if WITH_PREFIX did exist (or anything like it) SELECT table_a.* WITH_PREFIX 'table_a', table_b.* WITH_PREFIX 'table_b', table_c.* WITH_PREFIX 'table_c' FROM table_a JOIN table_b ON ... JOIN table_c ON ... ... So I just gave ChatGPT the schema definitions/query and it wrote out the long list of like 40 columns to be selected for me, like: SELECT table_a.id AS 'table_a_id', table_a.email AS 'table_a_email', ... table_b.id AS 'table_b_id', table_b.start_date AS 'table_b_start_date', ... and so on. I haven't found another good way to automate things like that across different RDBMSes (different queries for system tables that have schema information) and while it's possible with regex or a bit of other types of text manipulation, just describing the problem and getting the output I needed was delightfully simple. Aside from that, I just use the LLMs as autocomplete, which also encourages me to have good function naming, since often enough that's sufficient information for the LLM to get started with giving me a reasonable starting point. In particular, when it comes to APIs or languages I haven't used a lot, but the problems that I face have been solved by others thousands of times before. I don't even have to use StackOverflow much anymore. That's why I bought Copilot (though JS/HTML autocomplete in JetBrains IDEs is visually buggy for some reason) and use ChatGPT quite a lot. LLMs are definitely one of my favorite things, after IntelliSense (and other decent autocomplete), codegen (creating OpenAPI specs from your controllers, or bootstrapping your EF/JPA code from a live dev database schema), as well as model driven development (generating your DB schema migrations/tables from an ER model) and containers (easily packaged, self-contained environments/apps) and smart IDEs (JetBrains ones).
- ryanmcgarvey 3y agoMaybe it's worth reevaluating our definition of quality? In a world where AI can read our codebase, ingest a prompt, and quickly output "correct" if not clean and concise code, and then be able to iterate on code with more prompt, do we need all the same patterns we used to adopt when humans were painstakingly writing every line of code? This reminds of of the CISC to RISC migration - now that computers are in the loop writing the tedious parts, we don't need to burden our codebase with patterns meant to relieve humans from the tedium. I find myself, for instance, writing more long form, boring configuration files that once upon a time I would have built some abstraction reduce the boilerplate and verbosity. But now that co-pilot can just auto-complete the next section for me, why bother?
- noobermin 3y agohttps://en.wikipedia.org/wiki/Wirth%27s_law https://en.wikipedia.org/wiki/Wirth%27s_law
- IlliOnato 3y agoOld saying "what Intel giveth, Microsoft taketh away" :-)
- arp242 3y ago> In a world where AI can read our codebase, ingest a prompt, and quickly output "correct" if not clean and concise code, and then be able to iterate on code with more prompt, do we need all the same patterns we used to adopt when humans were painstakingly writing every line of code? That world does not exist, so currently this line of thinking is academic. Perhaps it will exist in the future, but it's far from a certainty if that will come to pass, and unclear on what kind of time-frame. Personally I'm quite skeptical any of us will see it within our lifetimes.
- patrickmay 3y agoPatterns aren't intended to "relieve humans from the tedium," they're to make the code more intelligible. Code generators create notoriously difficult to understand code. Apparently the current crop of LLMs are no better in that regard. > find myself, for instance, writing more long form, boring configuration files that once upon a time I would have built some abstraction reduce the boilerplate and verbosity. But now that co-pilot can just auto-complete the next section for me, why bother? Again, so humans can understand it more easily.
- spaniard89277 3y agoI'm a junior, and I have Codeium installed in VSCode. I've found it very distracting most of the times, I don't really understand why so many people uses this kind of assistants. I find stuff like Phind useful, in the sense that sometimes something happens that I don't understand, and 60% of the times Phind actually helps me to understand the problem. Like finding trivial bugs that I didn't spot because I'm tired, dumb, etc. On the other hand, with Codeium, I guess it may be useful when you're just churning boilerplate code for some framework, but in my little expericence (writing scrapers and stupid data pipelines & vanilla JS + HTML/CSS) cycling through suggestions is very irritating, specially because many times it doesn't work. Most of the times for stupid reasons, like lacking an argument or something like that, but then it's time you have to spend debugging it. Another problem I have is that I find there's a common style of JS which consist in daisy-chaining a myriad of methods and anonymous functions, and I really struggle with this. I like to break stuff into lines, name my functions and variables, etc. And so many times code suggestions follow this style. I guess it's what they've been trained on. Codeium is supposed to learn from this, and sometimes it does, to be fair. But what I worry the most is that, If I'm a junior and I let this assistants do the code for me ¿How the hell I'm supposed to learn? Because giving Phind context + questions helps me learn or gives me direction to go on find it by myself in the internet, but if the only thing I do is press tab, I don't know how the hell I'm supposed to learn. I found a couple days ago that many people (including devs) are not using LLMs to get better but it's just a substitute of their effort. Isn't people afraid of this? Not because companies are going to replace you, but it's also a self-reflection issue. Coding is not the passion of my life, addmitedly, but I like it. I like it because it helps me to make stuff happen and to handle complexity. If you can't understand what's happening you won't be able to make stuff happen and much less to spot when is complexity going to eat you.
- withinboredom 3y agoI think probably the best use of AI, so far, was when I went into a controller and told it to generate an openAPI spec ... and it got it nearly right. I only had to modify some of the models to reflect reality. BUT (and this is key), I've hand-written so many API specs in my career that 1) I was able to spot the issues immediately, and 2) I could correct them without any further assistance (refining my prompt would have taken longer than simply fixing the models by hand). For stuff where you know the domain quite well, it's amazing to watch something get done in 30s that you know would have taken you the entire morning. I get what you're saying though, I wouldn't consider asking the AI to do something I don't know how to do, though I do have many conversations with the AI about what I'm working on. Various things about trade-offs, potential security issues, etc. It's like having a junior engineer who has a PHD in how my language works. It doesn't understand much, but what it does understand, it appears to understand it deeply.
- dweinus 3y agoThe methodology seems to be: compare commit activity from 2023 to prior years, without any idea of how many involve Copilot. Then interpret those changes with assumptions. That seems a bit shakey. Also: "The projections for 2024 utilize OpenAI's gpt-4-1106-preview Assistant to run a quadratic regression on existing data." ...am I to understand they asked gpt to do a regression on the data (4 numbers) rather than running a simple regression tool (sklearn, r, even excel can do this)? Even if done correctly, it is not very compelling when based off of 4 data points and accounting for my first concern.
- zemo 3y agocheck out the paper, not just the summary. They explain their methodology. The output has four data points because it’s a summary. The input is … more data than that.
- lolinder 3y agoMore data, but OP is right on the weaknesses of the study—the author posted here [0] and acknowledged that they can't actually say anything about causality, just that 2023 looked different than 2020. [0] https://news.ycombinator.com/item?id=39168841 https://news.ycombinator.com/item?id=39168841
- TOMDM 3y agoI'd bet 2017 looked different from 2020 too.
- dweinus 3y agoI did, that's where my quote is from. The appendix confirms; they ran the regressions on just two inputs: 2022 and 2023 totals.
- gcr 3y agoYou can even see a truncated RankWarning in the output in the appendix. Low rank indeed! Numpy is crying out in agony and the authors didn’t notice They’re using GPT….to write code that does “quadratic regression” on two whole data points…which just extends the slope of each [2022, 2023] line one year further!
- graypegg 3y agoFeels like taking the average of all text on the internet, does result in barely OK output. Very few people are truly “average” in the numerical sense of the word, so you’re going to see better results from most devs compared to the coefficient soup that tries to find the commonalities between the best and worst devs. Truth be told though, speaking as someone that still does not use LLM tools at work… “just OK” is totally viable for a lot of things. Prototypes yes, products expected to be around a while, maybe not.
- withinboredom 3y agoI recently had this discussion at work. There's a difference between writing software that will be seen again in a few months and writing software that probably will never be touched again (or at least touching it again will be pretty risky). Identifying which one you're working on is super important. When you're writing software that is touched fairly often, the "tribal knowledge" of how it works will likely live-on. You can be a little bit clever at times, you don't need to comment as heavily, and your variable names can be a bit wonky. When you're writing software that is hardly ever touched ... everything needs to be crystal clear. Write lots of comments, explaining the "why" things are the way they are. You want to avoid cleverness except in the name of performance (but then comment how you expect it to work so that when someone comes along to change it, they understand wtf you are doing and the constraints you used to define the code). It's a totally different ballgame. AI doesn't get this distinction, hell, most programmers don't either.
- bemusedthrow75 3y ago[flagged]
- wowspeechless 3y agoYou're absolutely right. Why are they maligning contractors? Never mind the fact that the comment was insulting - it's not even factual. I'm hired as a contractor to fix broken projects.
- stakhanov 3y agoCouldn't agree more. Knocking on contractors seems random, out of place, factually inaccurate, and it doesn't really add anything to the substance of this article, so also completely unnecessary to sustain the actual point the article is trying to make. It's just bad writing.
- klyrs 3y agoGarbage in, garbage garbage garb garb ggggggggggg
- wbharding 3y agoOriginal research author here. It's exciting to find so many thinking about long-term code quality! The 2023 increase in churned & duplicated (aka copy/pasted) code, alongside the reduction in moved code, was certainly beyond what we expected to find. We hope it leads dev teams, and AI Assistant builders, to adopt measurement & incentives that promote reused code over newly added code. Especially for those poor teams whose managers think LoC should be a component of performance evaluations (around 1 in 3, according to GH research), the current generation of code assistants make it dangerously easy to hit tab, commit, and seed future tech debt. As Adam Tornhill eloquently put it on Twitter, "the main challenge with AI assisted programming is that it becomes so easy to generate a lot of code that shouldn't have been written in the first place." That said, our research significance is currently limited in that it does not directly measure what code was AI-authored -- it only charts the correlation between code quality over the last 4 years and the proliferation of AI Assistants. We hope GitHub (or other AI Assistant companies) will consider partnering with us on follow-up research to directly measure code quality differences in code that is "completely AI suggested," "AI suggested with human change," and "written from scratch." We would also like the next iteration of our research to directly measure how bug frequency is changing with AI usage. If anyone has other ideas for what they'd like to see measured, we welcome suggestions! We endeavor to publish a new research paper every ~2 months.
- lolinder 3y ago> That said, our research significance is currently limited in that it does not directly measure what code was AI-authored -- it only charts the correlation between code quality over the last 4 years and the proliferation of AI Assistants So, would a more accurate title for this be "New research shows code quality has declined over the last four years"? Did you do anything to control for other possible explanations, like the changing tech economy?
- oooyay 3y ago> We hope it leads dev teams, and AI Assistant builders, to adopt measurement & incentives that promote reused code over newly added code. imo, this is just replacing one silly measure with another. Code reuse can be powerful within a code base but I've witnessed it cause chaos when it spans code bases. That's to say, it can be both useful and inappropriate/chaotic and the result largely depends on judgement. I'd rather us start grading developers based on the outcomes of software. For instance, their organizational impact compared to their resource footprint or errors generated by a service that are not derivative of a dependent service/infra. A programmer is responsible for much more than just they code they right; the modern programmer is a purposefully bastardized amalgamation of: - Quality Engineer / Tester - Technical Product Manager - Project Manager - Programmer - Performance Engineer - Infrastructure Engineer Edit: Not to say anything of your research; I'm glad there are people who care so deeply about code quality. I just think we should be thinking about how to grade a bit differently.
- axegon_ 3y agoI'm not surprised. AI tools can be great at providing a quick, working example in simple scenarios, but it's just that: a quick(often dirty) working example. But I've seen people taking it as-is and putting it into a production codebase. Sure, a function that iterates over an array, and checks if some item exists - fine. In those cases it will do the job simply because you(it) can't get it wrong. However I had this experience where senior developers were fully invested into using it. And because managers see code just erupting like a volcano, they embrace it and in fact rise their expectation when it comes to delivering a feature: "you could do it in a week before, you should be able to do it in 2 hours now, right?". And on more than one occasion this has backfired massively in everyone's face. The last time me and another dev spent two straight days rewriting and debugging everything cause there was an international exhibition that was about to start and the company was at the front of the line and everyone else simply pushed a ton of code that was 75% AI-generated, completely ignoring the edge cases, which were more than people anticipated. But probably the most off-putting thing I've experienced is an OKR session with 50 people in it, where a lead dev publicly opened chatgpt, prompted "how do we reduce the number of critical bugs by 30% in the next quarter", chatgpt came up with a generic response, everyone said "perfect", copy-paste that into Jira and call it a day. And I'm just sitting there and wondering if there was something rotten in my breakfast and I'm hallucinating. Unfortunately my breakfast was fine and that really happened. The few times I've tried using those, they were only helpful with dumb, repetitive and mundane tasks. Anything else, you have to rinse and repeat until you get a working solution. And when you read it(for those of us that do), you realize you might have been better off recruiting a freelancer from year one in University to spend a day mashing it up and likely coming up with something better. But I bet much of those year ones would be doing this exact thing day in and day out until they come up with a solution: Occasionally I will grab my laptop and go work at a cafe on my personal projects for a change and I can't tell you how many times I've seen people just copy pasting stuff from chatgpt and pasting it back into their IDE/editor and calling it a day - students and clearly people who are doing this for a living. Not to mention copilot, that's the de-facto standard at this point. In fact I had this conversation last year with a guy(developer) who is 20-something years older than me(so mid 50-s): most of the LLM's are trained on stuff that is in the documentation, examples, reddit and stackoverflow. It's only a question of time until the content found in those exact locations where the training data is pulled from will become more and more AI-generated, models will be re-trained on those and eventually shit hitting the fan. I don't think we are too far off from this event.
- gnabgib 3y agoPreviously (same URL): [0](5 points, 2 days ago, 5 comments) [2](17 points) [3](9 points, 3 comments) "Poor code quality due to AI assistants GitHub Copilot and ChatGPT" [1](21 points, 2 days ago, 10 comments) [0]: https://news.ycombinator.com/item?id=39142285 https://news.ycombinator.com/item?id=39142285 [1]: https://news.ycombinator.com/item?id=39144366 https://news.ycombinator.com/item?id=39144366 [2]: https://news.ycombinator.com/item?id=39156643 https://news.ycombinator.com/item?id=39156643 [3]: https://news.ycombinator.com/item?id=39164079 https://news.ycombinator.com/item?id=39164079
- darepublic 3y agoSometimes less dry code can actually be easier to read and understand at the point of usage than dry code that has been more highly abstracted and requires grokking a sort of quasi DSL that defines the abstraction. Assuming that AI contributions will only increase, if a codebase were almost completely written by AI perhaps the benefits of DRY would diminish vs on the spot readability by humans only trying to understand the code and not personally add to it
- chongli 3y agoWell as with anything, DSLs are subject to the rules of good design. A really well-designed DSL (such as SQL) takes on a life of its own and becomes incredibly ubiquitous and useful to know in its own right. Many other DSLs are totally unknown, not worth learning, and serve as barriers to code understanding. I don’t know of too many people who would advocate replacing SQL with hand-written C manipulating B-trees and hash tables. Similarly, it’s pretty rare that you want to hand-roll a parser in C over using a parser generator DSL or even something like regex.
- rsynnott 3y agoShock horror, the thing everyone said would happen has happened. I mean, I’m not sure what people expected.
- andybak 3y agoSome people are thinking hard about a complex, nuanced topic of which we have very little past experience to draw on. I'm glad the conclusion is so self-evident to you. I must be a little slow.
- righthand 3y agoML/LLMs are nuanced and complex topics as they are the inner workings of automation. People using ML/LLM to get around having to write code already understood well enough isn’t a complex nuanced topic because it is the outer workings of automation and has been studied in other fields quite extensively. No one should be surprised at the trend of lazier development from wide adoption of automation tools.
- andybak 3y agoHow do past lessons about automation cleanly apply to AI code generation? Cleanly enough that there's no room for debate, doubt or discussion? (Edit. Not the same person. I keep making this mistake in discussion threads) Previously I wrote: > It's your tone of "this is obvious, people! Why are you still wasting time thinking about it?" that I'm taking exception to.
- righthand 3y agoThere’s room for discussion on it and where to go from here, it’s the result that is not surprising. I would say instead of reacting to the rhetorical remarks, bring up the actual interesting discussion around it in your response.
- danielovichdk 3y agoWhen I look into the future, and I know that I really can't, one thing I really believe in is that there will be a shift in how quality will be perceived. With all things around me there is a sense that technology is to be a saviour for many very important things - ev's, medicine, it, finance etc. At the same time it is more and more clear to me that technology is used primarily to grow a market, government, country etc. But it does that by layering on top of already leaking abstractions. It's like solving a problem by only trying to solvent be its symptoms. Quality has a sense of slowness to it which I believe will be a necessary feat, both due to the fact that curing symptoms will fall short and because I believe that the human species simply cannot cope with the challenges by constantly applying more abstractions. The notion about going faster is wrong to me, mostly because I as a human being do not believe that quality is done by not understanding the fundamentals of a challenge, and by trying to solve it for superficial gains is simply unintelligent. LLMs is a disaster to our field because it caters to the average human fallacy of wanting to reach a goal but without putting in the real work to do so. The real work is of course to understand what it is that you are really trying to solve with applying assumptions about correctness. Luckily not all of us is trying to move faster but instead we are sharpening our minds and tools while we keep re-learing the fundamentals and applying thoughtful decisions in hope to make quality that will stand the test of time.
- 0xedd 3y ago[flagged]
- staunton 3y agoPresumably, this was supposed to show how parent's arguments can equally well argue for the opposite or for something nonsensical. However, it seems incoherent to me (in a way that parent does not). I don't even know what it's arguing for... I guess something literally about sewing?
- c0pium 3y agoIt’s a style-transfer of OPs post onto Luddites. This same tired argument in 3 parts gets made every time a potentially job-destroying technology gets introduced or discussed. Since nobody has made the concluding argument yet, I’ll supply it; the Luddites were correct that it (mechanization) would destroy their lucrative profession, they missed the degree to which it would enable many other lucrative professions, and they were right that many Luddites would not make the transition. Society was net better off but there were definitely losers in the transition.
- kvonhorn 3y agoQuality is best thought of as a process, and that process got pushed out of the SDLC by Agile process and its metric of velocity. The use of LLM-generated code to further increase velocity in the absence of quality process is an obvious result.
- cleandreams 3y agoI am worried that AI assisted code will be a competitive advantage so that the downsides will not be addressed until there are serious failures in critical code. Boeing but for power plants, for example.
- mvdtnz 3y agoThere was already a backlash against DRY code occurring before "AI" assistants hit the market, sadly. It was a growing movement when I was using Twitter in 2019-2022. Some younger developers have a very different attitude to code than what I was brought up with. They have immense disdain for the Gang of Four and their design patterns (probably without realising that their favourite frameworks are packed to the gills with these very patterns). They speak snidely about principles like DRY and especially SOLID. And on places like Twitter the more snide and contrarian you can be, the more engagement you'll get. Very disturbing stuff.
- deadbabe 3y agoDRY is mostly bad. You couple a lot of things together by making them all share the same code and then a small change to that code breaks various unrelated components.
- abathur 3y agoI don't think it's mostly good or mostly bad in the abstract. DRYing code repeated for the same reason is mostly good. DRYing code coincidentally repeated for different reasons will sow code churn or inadvertent behavior shifts.
- deadbabe 3y agoIf the reason for DRYing up code is simply because it looks the same as some other code, that’s a bad reason.
- charles_f 3y agoDry's mostly a smell test to be applied when copying stuff over, to check that you're doing that for the right reasons, rather than expediting shit.
- osigurdson 3y agoFor sure, there are obvious egregious examples of repeating one's self. There are also many more examples of developers programming by unexamined catch phrases.
- happytiger 3y ago> We further find that the percentage of 'added code' and 'copy/pasted code' is increasing in proportion to 'updated,' 'deleted,' and 'moved 'code. In this regard, AI-generated code resembles an itinerant contributor, prone to violate the DRY-ness [don't repeat yourself] of the repos visited. Couldn’t this equally be explained by the cost of refactoring becoming incredibly cheap? If most of your code is generated, and you don’t have to make the investment of hand crafting everything, aren’t you naturally going to be regularly replacing vast tracts? Obviously the trend may have implications, but in large part aren’t we just seeing the impact of code cheapening? Serious question.
- userbinator 3y agoThose who can't code better than an LLM will welcome it, those who can will abhor it. Unfortunately it seems there are far more of the former than the latter, and they aren't going to get better either. Progress comes from human intellect, mediocrity comes from regurgitation.
- linsomniac 3y agoOn the other end of the spectrum, I'm finding ChatGPT is helping reduce the friction of adding unit tests, getting started with docstrings (I usually need to do a lot of work on them though), and type annotations. Plus I've had some luck with handing code to ChatGPT and asking it to suggest clearer ways of writing it. For example, a few months ago I was rewriting some particularly tricky, opaque code from Ansible having to do with the conversion of symbolic "chmod" strings into numeric. Because I was coming from code that was so hard to reason about (a bug in it took me a couple days to understand and fix), I wanted something really obvious and well tested. ChatGPT helped with that. https://github.com/linsomniac/symbolicmode/blob/main/src/symbolicmode/__init__.py https://github.com/linsomniac/symbolicmode/blob/main/src/sym...
- pylua 3y agoI think unit tests are suppose to induce friction to a certain extent. Using ai to do it seems like it is just checking a box.
- linsomniac 3y agoThe amount of boiler-plate, and in particular getting mocks right, are friction that can go away.
- web3-is-a-scam 3y agoI cancelled my subscription after 2 months because I was spending way too much mental effort going over all of the code vomit fixing all of the mistakes. And it was basically useless when trying to deal with anything non-trivial or anything to do with SQL (even when I frontloaded it with my entire schema). It was much less effort to just write everything myself because I actually know what I want to write and fixing my own mistakes was easier than fixing the bot’s. I weep for the juniors that will be absolutely crushed by this garbage.
- oldpersonintx 3y ago[dead]
- teaearlgraycold 3y agoReally? I've been doing web dev as a hobby for 20 years and professionally for 6 or 7 years. It's super helpful for me giving how much boilerplate there is to write and how much of the training set is for websites. Any time I try to get it to write non-trivial SQL or TypeScript types it fails hard. But it's still a nice time saver for writing tests, request handling, React boilerplate, etc.
- louthy 3y agoThis is the problem. As programmers we should be focusing effort on reducing boilerplate, so that it’s never needed again. Instead, we’ve created a boilerplate generator.
- CuriouslyC 3y agoThe challenge there is that reducing boilerplate comes with opinionated magic, which rubs a lot of people the wrong way. So the creators add configs and code level config, and pretty soon you're back in boilerplate hell.
- teaearlgraycold 3y agoNope. The wrong abstraction is too risky vs. having no abstraction. AI can let us move towards an abstraction-free mode of coding. In the future you’ll tell the AI “add a 60 second cache to every source of X data” and it’ll scour your source code for matching code and you’ll review the diffs. C programmers would review LLVM IR diffs. Web devs will review whatever compilation target they’re using. We use high level languages because they improve reading comprehension and save us time when writing. Having a copilot take on a major role allows us to fundamentally rethink programming.
- adonese 3y agoHave been using chatgpt/ copilot for a while now and I feel I know better the limitations and what it can achieve most: - unit testing and docs, readme - when it can potentially hallucinate It helps me do things that I'd usually procrastinate from doing, yet I know how I can get them done. It is really a booster to ones performance if you know exactly what you want
- phendrenad2 3y agoGithub Copilot is really good at writing spaghetti code. It isn't good at refactoring and modularizing your code. So if you're used to writing spaghetti code yourself, you can work with it and manage the complexity. If you're used to writing highly-factored code, the code it generates is going to require a lot of work.
- polyterative 3y agoI convinced my boss to pay for it I love it
- havaloc 3y agoUsing GPT-4 has significantly enhanced the efficiency of my work. My focus is primarily on developing straightforward PHP CRUD applications for addressing problems in my day-to-day job. These applications are simple, and don't use frameworks and MVC structures, which makes the code generated by GPT-4, based on my precise instructions, easy to comprehend and usually functional right out of the prompt. I find if I listen to the users needs I can make something that addresses a pain point easily. I often request modifications to code segments, typically around 25 lines, to alter the reporting features to meet specific needs, such as group X and total Y on this page. GPT-4 responds accurately to these requests. After conducting a quick QA and test, the task is complete. This approach has been transformative, particularly effective for low-complexity tasks and clear-cut directives. This process reminds me of how a senior programmer might delegate: breaking down tasks into fundamental components for a junior coder to execute. In my case, GPT-4 acts as the junior programmer, providing valuable assistance at a modest cost of $20 per month. I happily pay that out of pocket to save myself time. However, much like how a younger version of myself asked why we had to learn math if the calculator does it for us, I know understand why we do that. I think the same thing applies here. If you don't know the fundamentals, you won't be effective. If GPT-4 had been around when I learned to write PHP (don't @ me!), I probably wouldn't understand the fundamentals as well as I do. I have the benefit of learning how to do it before it was a thing, and then benefitting from the new tool being available. I also don't find the code quality to be any less, if anything what it spits out is a bit more polished (sometimes!).
- hackernewds 3y agoprecisely. we are very lucky to be during the timeline where ChatGPT was released during our later years, that we didn't have to compete with auto created code during our learning formative years.
- WanderPanda 3y agoThis sounds like motivated reasoning to me. Having an above-average personal tutor that doesn't get mad or tired 24/7 every time seems like a big multiplier
- tippytippytango 3y agoThe valuable thing to a business is not code, it’s the people in your business that understand the code. LLMs are ultimately going to be more valuable for reading code than writing it.
- whatever1 3y agoThe use of generative models will cause their own demise. At the end they will just produce pure noise.
- nestorD 3y agoI can definitely relate to that. I found that using ChatGPT to help me code led me to get working, well documented, code done much faster but it tend to also be somewhat more "naive" code: going for the obvious solutions even if their complexity is not optimal.
- superfunny 3y agoOne thing that I found especially helpful when using CoPilot and ChatGPT is that I can get an answer very quickly and directly, instead of having to wade through a lot of blogs or articles on Stack Overflow.
- hinkley 3y agoAnd a new generation learns the term, "overtraining". <hits the snooze button> Someone wake me up in fifteen years when the next batch of students repeat history.
- danpalmer 3y agoThis certainly fits with my experience and biases. When using Copilot, I felt that it tried to be too clever, and often got things wrong. It would try to write a function based on the name, and would go one of three ways: either the function was a trivial one-liner and it saved me ~2 seconds of typing after I figured out if it was correct, or it was a complex function where it cost me ~2 seconds to figure out it was waaaaay off the mark, or it saved me 30 seconds up-front by producing something that appeared correct, but that I found out 10 minutes of debugging later was actually subtly incorrect in a way that I couldn't spot when reading quickly, but likely wouldn't have written myself. What I really want is a smarter intellisense, whereas Copilot is a dumber pair programmer. I want smart, style-aware, context-aware tab completion on boilerplate, not on business logic. Unfortunately I think many people are using it for business logic and that seems to be the direction the product is going.
- myko 3y agoAfter using Copilot seriously for a few weeks, I can't believe they charge for it. It's really awful. It shows glimpses of being useful but rarely delivers usable code.
- aksss 3y agoIf all copilot did for me was autocomplete my exception handling routines, log messages, and class definition getters and setters, I would still pay for it. I think that's where it's competency really sits. It provides some interesting suggestions otherwise, but mostly it's these slam dunks that make me very happy with it.
- amiga1200 3y agoMuch like the Y2K bug kerfuffle, there will be a time when the Copilot bug will cause an upsurge in business requiring developers to fix the stochastic parrot code bugs. I think this will be sooner rather than later.
- shadowgovt 3y ago"'AI-generated code resembles an itinerant contributor, prone to violate the DRY-ness [don't repeat yourself] of the repos visited.' ... That serves as a counterpoint to findings of some other studies, including one from GitHub in 2022 that found, for one thing: "developers who used GitHub Copilot completed the task significantly faster -- 55 percent faster than the developers who didn't use GitHub Copilot." These two observations aren't mutually exclusive. DRY-ness comes from a holistic understsanding of a system; you get similar results when you task a junior developer with little deep knowledge of a novel codebase with solving a problem and don't give them enough oversight; they'll tend to bull-rush through coding a solution with little-to-no knowledge of how the solution can be built out of the parts already existing in the codebase.
- nf3 3y agoI've been watching from the sidelines this whole AI schtick with much amusement. I'm pretty sure it will end with a whimper.
- chilldsgn 3y agoI have a Jetbrains AI subscription and I use it mostly to write commit messages. It speeds up my work significantly, but I still have to skim over the generated output to make sure the message is correct. I tried it for code generation and it broke DRY multiple times. It's still faster for me to just build a thing without code generation. I'm not a wizard with my IDE, but I've gotten to a point where I'm pretty fast with some things I need to get done, it's muscle memory. I am a bit fanatic about verbose variable names, and sometimes the AI creates terrible names for variables, which I go and change anyway.
- itqwertz 3y agoAs someone with multiple projects, it helps with “good enough” or time-consuming tasks like filling out interfaces. basically it cuts down on context switch/flow state/ADD tendency time loss.
- freedomben 3y agoIt looks like the trend started heading upwards before AI took over. Are we sure this wasn't just the way it was heading anyway? Is there a control group that doesn't show the same line? Overall from personal experience TFA's conclusions do seem correct to me, but I want to make sure we aren't confirmation biasing the interpretation of the data.
- moi2388 3y agoCopilot is great, as is gpt-4. I use it as search engine, to give me pointers about parts of the docs I might’ve missed. I have never not modified and refactored the code before implementing it.
- eggdaft 3y agoI also see LLMs primarily as a better search engine. Stack overflow gave us code snippets and was a huge step forward. Now we can find those snippets and explanations much faster. The secondary use is as a duck to bounce ideas off. A duck that is not as smart as a human but always available. And I can ask stupid questions to check my understanding. The final use is for code generation. If I’m super tired it can do trivial coding for me. Or if it’s boilerplate, which is very little code for me. Generally it doesn’t help a great deal within the IDE, and I’m not completely sure it’s a net win there yet.
- freedomben 3y agoSomething I didn't see mentioned is also that over time this is going to feedback loop into the training set and compound (possibly exponentially). I.e. as more lower quality code hits github and is used for training, the output of the code will decline, which causes lower quality code to hit github, which causes output to further decline, etc. I am an experienced dev but new to ML so take with a grain of salt, but I really wonder if the future is going to be quality in the training sets rather than quantity. I have heard that the "emergent properties" don't seem really affected by bad data as long as the set is large enough, but at what point does this cease to be true? How do we guard against this?
- SlightlyLeftPad 3y agoThere’s a potential case study here. It seems that AI itself is vulnerable to misinformation which is something that might be glossed over in academic papers but has just started to be seen in the wild. I have used alternatives to Github Copilot that appear to be better about predicting my intent but I’m curious how rapidly that can turn around depending on the audience of the platform.
- firtoz 3y agoPersonally, I experience a great boom to my productivity. Sure, the code I get from copilot is not "the best", however it allows me to get so much more done. If it makes a mistake or two it's not really that big of a deal. However, I don't work in finance or "really important stuff", so ymmv.
- alkonaut 3y agoCoPilot is good a as a one line autocomplete. It's short enough that you can review the suggestion and decide whether or not to accept the autocompletion or type out your own completion. For reasoning about larger chunks of code I find ChatGPT better than CoPilot as an LLM assistant. Trying to use CoPilot for making large sections of boilerplate like the kind you might see in a db->api->web project is just full of frustration. It doesn't realize it makes makes tiny inconsistencies everywhere so you are permanently babysitting. I think the key takeaway is that if you have repeated code (An entity, a DTO, a controller, a frontend component all sharing some set of names/properties) then its better to change jobs than change tools.
- draxil 3y agoSurprise! As soon as you aren't thinking about your code and not understanding it...
- holliplex 3y agoThis makes sense to me -- I find that if I ask it to generate more than, say, a paragraph of code at a time[1], I'd be much better off just writing it myself. That said, I have issues with RSI, so every saved keystroke is extremely useful to me. I find Copilot's ability to accurately predict the next, say, ten words I'm about to type very useful. The usual insult thrown at LLMs is that they're "fancy autocomplete". I think that's actually about right, and I'm extremely sceptical about any claims made about their critical thinking skills which I think are essentially non-existent. Where I differ with most AI-sceptics is that I think fancy autocomplete is incredibly valuable. [1]: Let's say that's about a mid-sized function definition.
- deleted 3y ago[deleted]
- lobocinza 3y agoI used Copilot for a short period of time. It was super fun having it autocomplete code and most of the time and I just had to change a few things. This was on a codebase and language I was pretty familiar with so it just saved me a bit of time. The issue is that due to the law of large numbers some of the bad code it frequently produced found it's way to production. Eventually I caught myself debugging and fixing more issues introduced by Copilot than my own, issues that wouldn't make myself. But I think this was majorly an UX problem because often I triggered a completion without desiring or even noticing it due to typing fast. I was using Copilot in Neovim.