8 ms·
40% of the code GitHub Copilot users check-in is AI generated and unmodified
- codevark 4y ago[dead]
- luxurytent 4y agoI'm not surprised. A lot of the code I'll write in a day is trivial, calling helpers, writing test scaffolding, etc. I'm still building cool things the other 60% of time, but this job as we know it comes with a lot of repetition. (esp. if you write unit tests ;-))
- aliencat 4y agounit test definitely a great use of copilot.
- wokwokwok 4y agoWhat are you talking about? I don't know anyone who uses it for this; specifically tests are really bad if they're subtly wrong. Maybe to scaffold the test function, but the actual test if completely useless if you don't trust it. So like... generate code and have robust tests, or write robust code... but, it's really really daft to generate tests that might hallucinate some random crap (and copilot really does sometimes). Other people have said this, but copilot is auto complete. You use it every time you press tab. Do you accept an auto-complete suggestion 40% of the time? ...mmmm, yes, well, guess why 40% of code is generated by copilot. It's not because people are generating tests with it.
- rjh29 4y agoMost of the time I know exactly what tests I want to write but it's annoying to write them. Copilot autocompletes, I check it and then I use it.
- Waterluvian 4y agoFor me the worst part about this is that writing tests tediously makes me reach for layers of abstraction on my test code. And then suddenly my test code is complicated and needs its own test code, and changing a test can often be problematic to the abstractions I foolishly employed. Being able to churn out the boilerplate for tests, which can make DRY a non-concern, is great. I just hope that if I do this, I never get sloppy, and I always review the tests.
- rjh29 4y agoI agree! I try not to make test code too complicated, and prefer repetitive code over abstractions (else you have to test the tests!) so Copilot is useful there.
- dandellion 4y agoIn my case about 50% of the test code is just boilerplate, so I usually type the test name and generate the rest. Most of the time I have to rewrite the actual test logic (although very trivial tests are sometimes correct), but I probably keep more than 40% if we're talking tests specifically.
- paro_nej 4y ago> In my case about 50% of the test code is just boilerplate Why not replace it with a function call?
- aaomidi 4y agoSome languages are just noisy. Go, Java I’ve experienced are noisier than others.
- travisjungroth 4y agoA common occurrence is something like a pool of 10 actions where a bunch of tests each do 3 to 7 of them. This is very hard to abstract with a function call.
- lupire 4y agoThat makes the test harder to read. Tests should be dumb.
- dandellion 4y agoIn the case I was referring to, mainly to keep the code consistent with what was already there and the PRs small, as the code base is primarily owned by a different team. It's also pretty innocuous short tests that read well as they are.
- teaearlgraycold 4y agoI use it for unit tests. That's one of its best use cases. Obviously I read the code it writes down.
- kakwa_ 4y agoIn a recent personal C project (a custom archive file extractor), I have used chatgpt to generate tons of unit tests. And honestly, I was extremely impressed. With a bit of context, it was able to generate almost correct test data for a fairly complex binary data format. I've also leverage it heavily to: * generate doxygen comments * get usage examples of libs I never used * create a whole bunch of utils functions. And honestly for all these tedious tasks, it has done a far better job than I would ever had. Comments were consistent in styling, the base examples were fairly good, and the utils functions were well made, specially the error handling (which I would probably have semi-consciously skipped tbh). In fairness, I modified most of this code slightly to make it fit my project structure, or tweaked it a bit what the IA didn't quite get it right. (it never fully understood some offset fields in the file format, but got pretty close to at times. And in fairness my naming for these offset fields was a bit questionable). Heck, as a test, I even threw at it an RFC-like spec of the file format, and asked him to generate a python parser. The result was not 100% correct, but definitely a good start to iterate on. In the end, this side project took me 2 weeks to implement with chatgpt probably saving around 1 week of dev. It also greatly helped improving the quality of the project (better doc, better tests).
- bananapub 4y ago> I don't know anyone who uses it for this that would be surprising - it does a fantastic job of producing e.g. dumb unit tests. for instance, Copilot + a Go table testing template means you can churn out the simple "make a request to this url with this data, ensure I get a 200 and the response contains 'id' and not 'error'" extremely quickly. the code is trivial but tedious, so you can quickly inspect for sanity and run to ensure they pass, then commit and have them checking future changes. > it's really really daft to generate tests that might hallucinate some random crap (and copilot really does sometimes). variations on this comment are all over these threads, which is bizarre. hallucinating means you waste a few seconds reading the code it produced, not that you commit incorrect code. this isn't like ChatGPT advising people to drink bleach, Copilot is a dumb tool offering an expert (you) suggested solutions for your expert consideration
- wongarsu 4y agoI guess this could be true in some languages and settings, for example when a notable portion of your tests verify correct behavior with null values, arguments of wrong type, etc. When talking about unit tests that verify the actual logic, you should naturally become more careful with your copilot suggestions. Especially if copilot also wrote the code it's testing.
- cortesoft 4y agoYeah, to me this doesn’t feel that different than using a library from another developer, just slightly more customized. Most code that is run is not written by the final developer, but by people further up the chain.
- js8 4y agoThe real question is, why we need to write "trivial" code? I think people have lost the plot there. We should be writing better compilers and libraries, instead of using AI that will write more code to deal with. The obsession with code generation instead of creating proper abstractions is a scourge of today's programming. It's a regression in software engineering.
- layer8 4y agoI agree to some extent. But there are redundancies that are inherently beneficial and make code more robust and readable, like separating interface from implementation (where some declarative elements have to be repeated), or explicit type annotations instead of full-program type inference. There are also strictly necessary redundancies, like when you pattern-match an ADT, you repeat its declared structure. For desirable or necessary redundancies like that, AI code completion can still save time. Intelligent unit test creation (if it is indeed intelligent) is also a good use-case. Not to relieve the programmer from thinking, but to save typing and to point out potentially useful tests that you might not have thought of.
- nine_k 4y agoThis is trivial stuff which is handled well by any competent IDE from 2015, and even a well enough-working set of editor templates. I don't see where "AI" is required for stuff like this.
- js8 4y agoI agree, but also, this is actually where AI (whatever it is) would be helpful. Suggesting actual abstractions that could replace boilerplate. Of course, these can build on existing abstractions.
- lupire 4y agoThe boilerplate makes it readable by humans.
- never_inline 4y ago
- nickstinemates 4y agoA lot of the code it generates is trivial, or 1-2 lines of code that's obvious, at least for me. In which case, it makes no sense to modify it. I have a feeling this is the bulk of it.
- pirate787 4y agoThis is my experience as well. For longer code, Copilot introduces errors through making up variables and method names that don't actually exist on the object. These errors are very annoying but only 5% of the generated code.
- halfjoking 4y agoI tried Copilot last month and canceled my subscription because of all the small errors in generated functions. It was more painful to debug than write it myself. Maybe I should have lowered expectations and thought of it as improved code complete. I got spoiled by copy/pasting from ChatGPT. As it stands now I'd rather pay for a subscription to have the latest ChatGPT than for Copilot.
- shmoogy 4y agoI feel like it has been getting worse for me in the past few months. I will write a comment and it will autocomplete - but have something wrong - almost like using chatGPT to try to generate code. Mostly right - but not quite. Things it would have gotten in the past. Maybe it's just me.
- deleted 4y ago[deleted]
- micromacrofoot 4y agofor me it’s a fancy autocomplete at its worst! it’s a handy tool to cut down on repetitive tasks
- rabuse 4y agoThis is exactly how I use it also. It's great when it suggests the correct thing, but doesn't interfere with me if it's wrong.
- throw__away7391 4y agoSure this must be using the most generous possible interpretation of the data. In my IDE the suggestion is shown as an autocomplete option on basically every keypress. I will only hit tab to insert if it is what I was planning to type. Sometimes if I really want to use it I'll coax it, write a comment or edit my code a bit to give it some help until the suggestion is what I want. There's no way they're counting every autocomplete suggestion in the denominator of that percentage.
- lupire 4y agoIt's 40% of submitted code, not 40% of suggestions.
- throw__away7391 4y agoI guess you're right, I initially read it as "40% of AI suggestions don't get modified", not "40% of code is AI generated".
- jejeyyy77 4y agolol I use co-pilot and there is no way this stat is true.
- beebmam 4y agoIn my experience it's more like 80-90%, given the right prompts. Very little needs to be changed, usually
- jejeyyy77 4y agoI'd say closer to 40%. You almost ways need to adjust the parameters of function. Almost always need to rename variables, etc. And this is writing mostly boilerplate too. For more custom stuff, I mostly just ignore the suggestions since they are mostly useless.
- sebzim4500 4y ago> 40% of the code GitHub Copilot users check-in is AI generated and unmodified > there is no way that stat is true > I'd say closer to 40% What?
- bee_rider 4y agoTechnically 0% is closer to 40% than it is to 90% although it would be weird phrasing.
- ratg13 4y agoI think it’s more interesting that they track this. I’d be more interested to see the other stats.. How much code is AI generated and then modified. How much modification is typically needed. How often suggestions are not used/abandoned/deleted.
- taf2 4y agoI just added an Ask AI feature to our product to help customer who want to write inline javascript functions to modify workflows with code instead of UI... so far it generates amazingly good code with a realtively simple system prompt.. that just let's the AI know ahead of time details about our API such that when a customer says: "Write a function to search for a lead in salesforce with jsforce and update an activity record with the duration of the call." It can really generate code that does just that... I do have to tell it to use a specific jsforce library or it'll just generate non working code... but IMO that's unreal and amazing... That question generated this code: [I did not write anything beyond this point] Here's an example function that searches for a lead in Salesforce using jsforce and updates an activity record with the duration of the call: ``` exports.handler = async (event, context) => { const jsforce = require('jsforce'); // Create a connection to Salesforce using the access token and instance URL const conn = new jsforce.Connection({ instanceUrl: event.salesforce.instanceUrl, accessToken: event.salesforce.token }); // Search for a lead with the given caller number const lead = await conn.sobject('Lead').findOne({ Phone: event.activity.caller_number }); if (lead) { // If a lead is found, update the activity record with the duration of the call const duration = 10; // Replace with actual duration of the call await context.ctm.update({ activity: { id: event.activity.id, duration: duration } }); } else { console.log('Lead not found'); } }; ``` Note that this function assumes that the caller number is stored in the Phone field of the Lead object in Salesforce. You may need to modify the code to match your specific use case. Also, the duration of the call is hardcoded to 10 seconds in this example, but you should replace it with the actual duration of the call.
- weq 4y agoCopilots positioned to optimise what i call "Google developer" flow. These are commadonity devs every workplace has who dont really know how to code, but rather know how to ask google questions to get the code they can slap together to meet a AC. On the other side you have devs who have AI in their head. They often use tools to optimise their flow. Ive always used resharper as a developer and think autocomplete is certainly a developer efficiency improver. 99% of the suggestions resharper makes, i checkin. Thats why i use it, it has less fuzz because it doesnt try to be everything. Pressing alt+tab and scrolling through some search results isnt something that adds overhead to my flow - after 30yrs im pretty good filtering content. I do alot of PoCs though, so if i wanted to write in a language i didnt understand, i think copilot can improve my flow somewhat to get me to serach terms that will benifit me quicker. Overall im still a few years away from relying on AI to help me code. Its going to be great for retirement though, if i had a family and kids it would allow me to checkout of self-paced learning while still maintaining my edge.
- ojkelly 4y agoSo far I’ve found copilot is like a weather forecast. The shorter the prediction the higher probability of it being correct. The longer, and more complex the prediction the less likely it’s all correct. But, with a close review you can spot the errors and correct it. I’ve found most useful to get ideas on the page fast. It’s also absolutely insane to be writing a parser, and have it suggest unit tests and code where it clearly has developed an understanding of the language before the parser is complete.
- ImPostingOnHN 4y agothe generalization is errors compounding over time
- markasoftware 4y agoYou bring up an important point in general: Copilot is great at writing tests.
- ALittleLight 4y agoI'm thinking about this in the context of GPT-4 going from bottom 10 percentile to to 90 on the LSAT and similar standardized tests. What's going to happen to this figure when they get bigger and better models to drive Copilot?
- r3dneck 4y agoIMHO, if you really "tell a CEO I can give him 56% more developers overnight" , he will sue you. Every programmer tried it out can tell that his affirmation could be right only for trivial part of a program, for non trivial programming part, it is more like a board game, where you try to induce the "AI" to produce the right answer modifying the "prompt" again, again, again, again, ( Did I tell you "again" ?) and again. <sarcasm> Obviously that is a customer fault because he didn't hire prompt engineers to do properly the job to describe properly the job they need be done </sarcasm>
- aaomidi 4y agoI mean this is a prime example why lines of code are a terrible metric.
- monero-xmr 4y agoThere is a lot of annoyance in the boilerplate of a program - "set up a scaffold for this, template that, convert data from this format to that format". The AI tools I have experimented with are good at this and for sure this is useful. But no one can read my mind! If I have to get increasingly specific in what I am telling the AI, and have to formulate my writing style in a way that keeps things very precise, I am essentially programming in another language. Albeit one where the output is uncertain and I have to check everything. It is possible that someday it reads in an entire codebase (say, millions of lines of code), and I can describe a new API I need and it implements the entire thing. This would be quite useful.
- Swizec 4y ago> It is possible that someday it reads in an entire codebase (say, millions of lines of code), and I can describe a new API I need and it implements the entire thing. This would be quite useful. You can get this today by switching to being a product manager/owner! Dream up an idea, describe it to a bunch of fuzzy intelligences, go through a few cycles of refinement, and voila: A software.
- rossjudson 4y agoAnd now on to the other 99% of the problem. ;)
- xtreme 4y agoOnly problem is that they cost $10,000/mo instead of $10/mo.
- lupire 4y ago$/mo doesn't matter. $/function does.
- deleted 4y ago[deleted]
- paxys 4y agoThat was my exact thought. I don't use copilot but I'd wager that at least 40% of the code I check-in is autogenerated as well. A fresh modern JS project is 15+ files full of boilerplate around a single "hello, world" statement. If there was a metric for how much net new application business logic was written by copilot, that would be far more interesting to study.
- mtlmtlmtlmtl 4y agoRead: 40% of the code Github Copilot users check-in is either boilerplate or has bugs in it.
- ascotan 4y agotrue that. 40% of the code is AI generated doesn't mean any of it actually runs. Morely likely we're on the hype cycle of AI generated code right now.
- symlinkk 4y agoAll of you should stop posting your knowledge of programming online (Hacker News, Stack Overflow, Reddit, blogs, etc) unless you want an AI to absorb it and take your job from you.
- steve1977 4y agoI guess they should primarily stop using GitHub as a source repository.
- bg24 4y agoI am a PM and a novice programmer. Unless copilot improves drastically for me, I am going to cancel the subscription and keep the one for chatgpt pro for now. I have been able to write fairly decent programs using chatgpt, am starting learn and build full stack apps etc. I am not able to use copilot. Maybe I do not know the tricks.
- alphabettsy 4y agoYou have to get Copilot started, writing a comment is often enough to do so. It doesn’t write from prompts like ChatGPT of course.
- fungiblecog 4y agoThe boilerplate code problem was solved by Lisp in the 1970s but we keep insisting on coming up with worse solutions because ‘parens look funny’
- nine_k 4y agoMostly because macros are unintuitive. But look, higher-order functions went mainstream a few years ago; immutable data structures, too. Sum types claw their way into mainstream, first to replace null, then for error handling. The industry is not hopeless. (are (excessively funny) (serialized (of (parse tree) structures)) though) ;; this limits direct Lisp adoption. Python and JS / TS took its place now.
- AlchemistCamp 4y agoI wonder if it’s 40% of the modules,40% of the functions, 40% of the lines or 40% of the tokens. The first would be very impressive, the last not so much.
- bhawks 4y agoSo how long before exploiters train code completion ais to generate back doors in the mountains of other boilerplate?
- diceduckmonk 4y agoHow is this different from Google searching then copy and pasting from Stackoverflow? Software is hard because of the 20, not the 80.
- never_inline 4y agoA. Direct solution vs choosing from 4-5 stack overflow answers. Gives an illusion of doing your work for you. B. AI can slightly manipulate existing code according to your prose. But I have also seen it fail hilariously to follow basic instructions. Eg: without `All argument constructor` in - but it still generates the all argument constructor. Another day it writes regex but fails to escape some characters properly.
- specproc 4y agoI had it on yesterday and had the most miserable afternoon's coding I can remember for a while. It was a particularly tedious, ugly, and cludgy bit of work that needed doing. Pulling in a data from several very broken sources with horrible naming conventions, patching holes and generally trying to make a sane dataset out of nonsense. I'm convinced Copilot made the whole process ten times harder than it needed to be. It was constantly pre-empting me with plausible-sounding variable names and merges which led me off down the wrong path a few times. I flick between having it on and off, and recently I've found I'm much happier with it off. Great for docstrings, boilerplate, and stuff like "recursively traverse this json for [key] and return [value]", but for day to day stuff it's a pain in the arse.
- ok_dad 4y agoSo let’s have a test where two or more teams build the same software. One set has AI tools, the other doesn’t. The software definition is provided to both at the same time, and there is a set of tests to see which team produced the software the fastest and produced the highest quality software.
- sph 4y agoAside: am I the only one here that is still writing code by hand and not using AI to autogenerate it? Because it seems over the past couple years on HN everybody is making great use of Copilot and constantly talking about it, while honestly scaffolding boilerplate is the least of my problems, while thinking, designing and abstracting novel ideas into code is the hard part where AI would fall short and just waste my time. I still do not get if I'm behind the times or just experienced enough (I've been programming for two decades) that I don't feel the need to use Copilot. It's a bit disconcerting because I feel I need to get out of pure programming work ASAP as everybody else is competing with me with Copilot and GPT and I am feeling an existential risk to my craft and livelihood.
- zarzavat 4y agoCopilot is way more than scaffolding boilerplate, it catches bugs as you are writing them. For example you expect: x[i] But copilot suggests: x[i+1] …and you think, and you think, and then one of two things happens. Either you satisfy yourself that the answer is indeed x[i], or you accept x[i+1] and sheepishly thank Copilot for catching a bug today. You could have copilot on in the background and never accept a suggestion, and it’s still a productivity enhancement.
- nathansherburn 4y agoI found during my preview months ago that it would often write bugs that took me longer to correct than if I had written it myself. Has it gotten better? Also in the situation where it suggests x[i+1] and it's wrong, doesn't that annoy you? In both cases I'm sure this is just temporary while LLMs get better. But my experience was they were _just_ a little too annoying last time I used them.
- bananapub 4y ago> Has it gotten better? yes
- markasoftware 4y agoIndeed, it introduces subtle bugs related to indexing pretty often. At one point I had it autocomplete maybe 15 lines of code, and I trusted it because the code was all pretty easy and I had written a descriptive comment, which tends to help out Copilot a lot. However, it introduced a very subtle bug where it used a loop counter `i` where it should have been indexing into an array like `arr[i]`, but it still typechecked and happened not to cause insane errors. Took a lot of time to track down later, and it's a bug I never would have written in myself.
- ElCapitanMarkla 4y agoNot sure if I’m using it wrong or the problems I’m coding for aren’t totally generic. The best help I get from it is a little string formatting or some basic error handling etc. I actually cancelled my subscription earlier today because I’m not getting enough benefit from it.
- miga 4y agoThis is double risk: 1. Security vulnerabilities from unaudited code. 2. Copyright violation risk, since a lot of CoPilot examples show clear excerpts from open source repositories.
- not_me_ever 4y ago99% of that code is stolen. Well done Micro$hit.
- johnrushx 4y agoAI is definitely going to revolutionize the industry! I'm excited about replit, marsx.dev, and github copilot.