28 ms·

GitHub Copilot Generated Insecure Code in 40% of Circumstances During Experiment

261 points by elsombrero 5y ago

gfiorav 5y ago
I think this should be pretty much expected. I'm unfamiliar with how this network is trained, but I'm pretty sure the data ranking is not perfect. I'm guessing the ranking features are based on the repo stats, contributor stats, etc. Even "good" contributors will make rookie mistakes in certain areas. Interesting to imagine how GH will try to solve this issue.
- jpalomaki 5y ago
  It might be possible to learn from the change history of the projects. There's likely quite many commits which fix certain security issues, such as SQL injection problems. Maybe even with suitable metadata in the issues or commit messages.
lmilcin 5y ago
I thought this should have been expected. Security starts with deep understanding. Some standards and practices can help avoid some types of problems, and some are even rather effective (like airgapping your systems), but there isn't any way to assure security in general other than truly understand what you are doing. ** I feel like Copilot is the wrong direction to optimize development. This is mostly going to help people with already poor understanding of what they are doing create even more crap. For a good developer those low level, low engagement activities are not a problem (except maybe for learning stage where you actually want people engaged rather than copy/paste). What it does not help is the important parts of development -- defining domain of your problem, design good APIs and abstractions, understanding how everything works and fits together, understanding what your client needs, etc. Also, I feel this is going to help increase complexity by making more copies of same structures throughout the codebase. My working theory about this is this is going to hinder new developers even more than they already are by google and stack*. Every time you are giving new developers an easier way to copy paste code without understanding you are robbing them an opportunity to gain deeper understanding of what they are doing and in effect prevent them from learning and growing. It is a little bit like giving answers to your kids homework without giving them chance to arrive at the answer or explaining anything about it. ** Another way I feel this is going to hurt developers is competition in who can produce most volume of code. I have already noticed this trend where developers (especially more junior but aspiring to advance) try to outcompete others by producing more code, close more tickets, etc. Right now it means skipping understanding of what is going on in favor of getting easy answers from the Internet. These guys can produce huge amounts of code with relatively little actual engagement. To management (especially with wrong incentives) this seems like a perfect worker, because management usually doesn't understand the connection between lack of engagement and planning at design/development time with their later problems (or they don't feel it is them that is going to pay the price). The Copilot is probably going to make it even more difficult for people who want to do it the right way because even starker difference in false productivity measurements.
- bsenftner 5y ago
  I too feel this is the wrong direction, from the fundamental aspect that trained algorithms contain no comprehension of what they are doing. They are the classic idiot savant. In a technological economy, comprehension of the environment is everything. I do not see how comprehension can be achieved without the elusive General AI, so I do not see this as anything other than a new area of research exposing how vitally important it is to have comprehension.
- andix 5y ago
  The real evil here is boilerplate code. I've seen so much boilerplate in the Java or classic .NET Framework world, it's incredible. So many layers of DTOs, Request/Response Models and so on, that could be just generated. Or most of the time even removed completely (that would cost some "architects" their job though). This is also true for a lot of Redux or Angular/NgRx applications. So much boilerplate, that you can't find the relevant code anymore.
  - fn1 5y ago
    This is because "Copy'n'paste" programming get's more and more common. I see more and more juniors pasting code or shellcommands from StackOverflow with careless ease, without even pretending anymore that they're interested in how it actually works.
  - lmilcin 5y ago
    (I have been professionally programming Java backends for the past 16 years). Java is not the culprit here. I think it is something that happened on the way that has something to do with J2EE and patterns craze we had a decade ago or two ago. It doesn't help that frameworks like Spring and their documentation go out of their way to propagate these boilerplate-heavy patters. Copying these lazy patterns is shortest, easiest way to get to working solution for a person that doesn't want to put any extra effort. And you can't get punished for doing this. Most developers don't even know there exist any other possibilities than mandatory controller calling service calling database layer and hordes of DTOs some people call "model".
wccrawford 5y ago
I'm actually impressed with that. There's so much insecure code out there that I'd have expected it to generate insecure code most of the time. I'd still not use it. But it's an impressive trick.
mbrevda1 5y ago
For comparison, what percentage of human-generated code is secure?
- westurner 5y ago
  > For comparison, what percentage of human-generated code is secure? Yeah how did they measure? Did static and dynamic analysis find design bugs too? Maybe - as part of a Copilot-assisted DevSecOps workflow involving static and dynamic analysis run by GitHub Actions CI - create Issues with CWE "Common Weakness Enumeration" URLs from e.g. the CWE Top 25 in order to train the team, and Pull Requests to fix each issue?: https://cwe.mitre.org/top25/ https://cwe.mitre.org/top25/ Which bots send PRs?
- iainmerrick 5y ago
  It seems reasonable to want Copilot to help you produce code of a reasonable quality. If it’s just helping you crank out the same bad code more quickly, without learning anything in the process, that’s useful to know. Some people might still want a tool like that, I wouldn’t.
  - burnished 5y ago
    Sure. But in order to know if its 'of reasonable quality' you need some sort of baseline to compare it to. What is reasonable quality? I think what your average human does is probably reasonable. Like, if your average dev will produce insecure code in 80% of samples, then Copilot starts to look really good! But if its closer to 0.01% of code samples, then copilot looks more like an intriguing novelty, not to be brought too near serious work. Much like dippin dots in this regard.
- eddieroger 5y ago
  That's basically where my gut went when I read the headline - so is that of a junior engineer, or really any engineer who hasn't had to think about it, and we don't promote their code directly to prod, either (if we can avoid it). Copilot shouldn't be able to generate code destined for prod without review any more than should any line of code written by a human.
luke2m 5y ago
So what? Copilot isn't intended to replace humans, just help them out and maybe reduce typing.
toastal 5y ago
You are the free labor copilot to train Microsoft GitHub's Copilot tool. You are responsible for any of those insecure code errors and the diligence require. You will be on the hook for resulting problems. But Microsoft and their home-phoning, tracking-embedded editor will get real people to correct and train their machine for free—with their stated plan of later selling that machine back to us later. I wish there were a “robots.txt” file for Git to disallow certain bots from training on anything I have written.
- gfiorav 5y ago
  Plus, GH doesn't care how you licensed your code. It will learn from it and produced licensed code.
  - deleted 5y ago
    [deleted]
- carl_dr 5y ago
  > I wish there were a “robots.txt” file for Git to disallow certain bots from training on anything I have written. It’s simple. If you are concerned by this, don’t host your repositories on GitHub.
  - tyingq 5y ago
    You would have to not host your code publicly either, right?
    - nextlevelwizard 5y ago
      You can always host is with license that doesn't allow reuse or something
      - tyingq 5y ago
        GitHub mentions that they don't currently look at the license before trawling code. https://twitter.com/NoraDotCodes/status/1412741339771461635 https://twitter.com/NoraDotCodes/status/1412741339771461635 There's also other references that GitHub public repos weren't the only source. They trawl other publicly readable code.
evolveyourmind 5y ago
Meaning 40% of the code on GitHub is insecure
- auggierose 5y ago
  No. It means that when c = f(a, b), where a, and b are secure, and you have no clue what f does, it might still be the case that c is insecure.
- dkersten 5y ago
  You could train a model on purely secure code and still have it combine it in insecure ways.
  - 0-_-0 5y ago
    And the other way around.
    - SketchySeaBeast 5y ago
      I guess the question really is how many ways are there to do it wrong, and how many ways are there to do it right?
moretti 5y ago
I use Copilot mostly as replacement for intellisense and macros. It helps me automating repetitive tasks. I would never trust Copilot for an algorithm or a snippet, I mean I would treat the code just like anything taken from StackOverflow or Github.
lvl100 5y ago
Why would anyone use this in production? Just use Sourcegraph if you need help that badly.
mullikine 5y ago
But this problem is solved using GitHub's CodeQL for searching and filtering generated code. By combining Copilot with GitHub Semantic and GitHub CodeQL, you have a means of writing and generating the code you want in a secure way. This means that you no longer need the original source code that was used to train Codex. Training Codex and selling as a product in the form of Copilot steals the essence of the original source code used to train it, to build the future of programming, while paying nothing back to the original authors. Even Elon Musk was opposed to OpenAI exclusively licensing to Microsoft GPT. https://edition.cnn.com/2020/09/27/tech/elon-musk-tesla-bill-gates-microsoft-open-ai/index.html https://edition.cnn.com/2020/09/27/tech/elon-musk-tesla-bill... It's so transformative that people may allow it to circumvent licenses.
gnrlst 5y ago
I've experienced this first hand: the autosuggest is scarily accurate and insidious at the same time. On numerous occasions I've auto-filled a 10-15 line suggestion that looked like it was exactly what I wanted to do, but made a very critical mistake (e.g. in a For loop, referencing the wrong array despite calling it the right name). Not really security related stuff, but head scratchers that make it harder to debug since I didn't actually write the code.
makach 5y ago
..does that means we will be 60% more secure than before?
- dkersten 5y ago
  Only if 100% of the code was insecure when written by a human. Given that humans can think through what code is doing, I don’t think that’s a reasonable assumption.
monkeydust 5y ago
I think where OpenAI Codex (which is what Copilot uses) gets more interesting is when the allow you to fine-tune the model on your own (trusted) code. That could help reduce the time it takes for new engineers to get up to speed for example.
bottled_poe 5y ago
What’s the baseline? 60% may still be superior to the average implementation.
lampe3 5y ago
I'm using copilot now for some time and yeah it's more a toy than real help right now. The only time it really helped when I needed to create a named list of char codes. When it comes to more complex code than checking the code of copilot takes the same time as writing it. 90% of the time I needed to correct copilot. For me, tools like linters are way more helpful then. If I could only use ESLint or copilot, I would go 100% of the time with ESLint.
- harlekein 5y ago
  I think another risk with getting Copilot to start out, is that it might nudge you into a direction you wouldn't have gone into otherwise. Whether that is better or not, I suppose, it depends.
  - lampe3 5y ago
    I'm working on a HTML Tokenizer in Deno/Typescript. Copilot only helps with boilerplate code which could be handled by good intellisense. When it tries to generate a function from the function name it fails so hard that it is more in your way then helpful.
0-_-0 5y ago
I fail to see how this is particularly useful information about Copilot. The comparison should be: 1. How many times do people write insecure code when not using Copilot? 2. How many times do people write insecure code when using Copilot?
- nextlevelwizard 5y ago
  It is useful since it means copilot is not taking your job any time soon. i.e. if 40% of the time the human driving the thing is needed to intervene and prevent obvious security flaws then expert is still needed to use the tool.
  - 0-_-0 5y ago
    I think it was obvious from the beginning that it's trained on GitHub code, so it would be surprising if it was better than the average code on GitHub. In any case, if Copilot can generate code as well as the average programmer without supervision, that means it can already take the job of 50% of programmers. A more useful metric though is how many programmers can a person using Copilot replace by having greater productivity? Also, in how many programming jobs does security matter? In my job for example it doesn't matter at all.
    - nextlevelwizard 5y ago
      With any commercial software security matters. You don't want your licensing solution to be craked by editing a file. Anything that connects to any server or has any sort of networking. Anything that needs privileges (like installing drivers) needs security. Anything that reads/interpeds any data given to it needs to have security. Of course this all depends what you mean by "matter". I can't come up with a program that doesn't need to think about security at all expect something trivial like hello world.
eurasiantiger 5y ago
I wonder if the Copilot model could somehow be repurposed to analyze the quality of a developer’s code. Seeing how Microsoft owns both GitHub and LinkedIn, it’s a good bet this is something they’re actively researching.
shireboy 5y ago
…Compared to 60% of circumstances in the meat-based developer control group? :)
- spywaregorilla 5y ago
  and of the population that is likely to use copilot in production for their own work? 90%?
  - lupire 5y ago
    These are made up numbers. A control group is needed.
- dimitrios1 5y ago
  I love that we always use the average here for these justifications. We just slowly chip away and any and all excellence. 10x memes aside, we all know what it's like to work with a truly talented and productive engineer versus your everyday schmoe collecting a paycheck. It's a story as old as time, and yet here we are doing the exact big factory industrialization techniques other industries have done and that is commoditize the thing that made them exceptional and eliminating artisanship, uniqueness, and ultimately quality and character. It's a tragedy of the commons of a sort.
  - phreeza 5y ago
    It may be a tragedy, but I fail to see why it is a tragedy of the commons? Which resource that is a available to all is being overused? High-paying dev jobs? Those are not a commons in the sense that tragedy of the commons implies because lower-quality devs don't stand to benefit by only taking a smaller part of the job.
    - runnerup 5y ago
      This actually feels to me like a concept worth exploring. I think we lack a concise term or phrase to reference what GP was trying to communicate. In my heart I feel similarly to GP - and it does feel a lot like how I feel about tragedy of the commons situations. Maybe there seems to be a shared opportunity for everyone if these private companies would make the most of their financial capital, market dominance, dominance in human resources, and most especially leverage their network effects. That would lead to better things for everyone, like the invention of smartphones. But the same corporations can also waste unimaginable resources and achieve very little. Often their failures don't just have little effect, but rather the failures choke/smother the market and prevent better alternatives from being widely used.
wcarss 5y ago
I couldn't find a link to the actual study anywhere in the article: https://arxiv.org/abs/2108.09293 https://arxiv.org/abs/2108.09293
lupire 5y ago
The thing about copilot that drives me crazy is that it is anti-good-programming. A smart AI should recommend a library function to call, not generate boilerplate code. The boilerplate code is either wrong, or should be a library function, nearly axiomatically.
arvindamirtaa 5y ago
Unit tests with TONS of assertions, cleaned data from form to ORM object, stuff that look look like you're just through a list and doing the same thing over and over. For these, Copilot is great. I wouldn't trust it to do anything else though. Nothing more. Nothing less.
softwaredoug 5y ago
I will say I’m not looking forward to writing some mundane code today. It’s interacting with GCS to scan a bucket for an extension, load the data with pandas, and concat some dataframes. It’s something dumb but mildly finicky that’s going to eat up so much time I could be using for higher value work. Copilot would be very welcome as I do this, instead of annoyingly going off to Google 3 different python libraries and getting it all to work nicely together.
ransom1538 5y ago
Has anyone here got past the wait list? I and my team members have been waiting for months.
queuebert 5y ago
This is exactly what an AGI would do if it wanted to pwn all our systems.
Vaslo 5y ago
It's learning from existing code, right? Doesn't this say something about developers in general, or is the thought that it uses combinations of code that are insecure?
- harlekein 5y ago
  I don't hold the average developer in very high regard. There are tons of developers who are much better than me and I readily read their books, follow their tweets, blog posts and online talks to learn from them. I hold them in high regard, but these people are not the average developer. If you would pick any smaller company with a dev team, a freelancer or an agency, your chances of finding a developer who understands and upholds quality code is vastly reduced. Not to mention a lot of beginners will just push their practice projects to GitHub and never look at it again. I'm also guilty of this, but I never realized Microsoft was training AI with this code. If Copilot is learning from these projects then I'd say the code it regurgitates is not average, but even below average.
cblconfederate 5y ago
Assuming that this is what it learned from its human counterparts, i'm surprised it 's so low.
rcarmo 5y ago
As many people have pointed out indirectly, this is almost certainly caused by the training set. Without a bias or ranking for quality, it will just churn out the “best fit” or most popular snippets…
amw-zero 5y ago
If it's trained on code that we write, that sounds completely accurate.
whazor 5y ago
Happily having access to GitHub copilot, it very often generates the code that I want. So it saves me from typing and also often saves checking Stack Overflow. I think the libraries/packages you use also play a big influence in how easy it is for copilot to create security flaws. Still, more training against security holes would be appreciated.
mzs 5y ago
the actual paper: https://arxiv.org/abs/2108.09293 https://arxiv.org/abs/2108.09293 previous discussion including comments from lead author: https://news.ycombinator.com/item?id=28279365 https://news.ycombinator.com/item?id=28279365
adamsvystun 5y ago
It is important to remember that Copilot can improve. 40% is not a bad baseline, but one data point does not give us much info, we should wait and see the rate of improvement.
dexen 5y ago
Half joking: so far GitHub Copilot is more feasible as tool for humans doing code-coverage for its input code, "given enough eyeballs, all bugs are shallow" style. When a developer goes, "huh, Copilot generated insecure code, better report it to the original project it learned it from" - if only Copilot was able to link to the original project, it would all be great and useful.
Animats 5y ago
Well, of course. GPT-3 has no underlying model of meaning. It's just autocomplete with a bigger data set. Used on natural language, it produces text that looks reasonable for about three paragraphs. Then you realize it's just blithering and has nothing to communicate. (Like too many bloggers, but that's another issue.)
cannabis_sam 5y ago
Of course it did, why would github copilot “care about” security, unless the majority of code on github cared about security?
COMMENT___ 5y ago
It's painful to see that GitHub Copilot is called "AI". For god's sake, it is not AI. It's just an advanced auto-complete for coders. GPT-3 is close to AI, GitHub Copilot is not. Jesus Christ, please make them stop. Stop using AI as a buzzword.
- sprafa 5y ago
  Is it fair to call it an ML based system?
  - COMMENT___ 5y ago
    Why not? GitHub Copilot was trained with ML to autocomplete your code. But it is not AI.
- arthur2e5 5y ago
  GitHub Copilot is literally GPT adapted for code. The paper on OpenAI Codex, the stuff powering Copilot, makes it very clear in the abstract. https://arxiv.org/abs/2107.03374 https://arxiv.org/abs/2107.03374 Either you call both AI or you call neither AI. (A previous version of the comment stated that it was tuned from GPT-3. This is incorrect; the simpler GPT was used for faster convergence.)
  - COMMENT___ 5y ago
    If I have to choose, then I would decide not to call them AI at all.
IlliOnato 5y ago
My take on Copilot has not changed. I believe it will make programmers that produce junk code more productive, by being able to produce more junk code in less time.
RhysU 5y ago
See also... You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completion https://deepai.org/publication/you-autocomplete-me-poisoning-vulnerabilities-in-neural-code-completion https://deepai.org/publication/you-autocomplete-me-poisoning...
spyder 5y ago
Well, it wasn't trained to output secure code was it?

261 points by elsombrero 5y ago

I think this should be pretty much expected. I'm unfamiliar with how this network is trained, but I'm pretty sure the data ranking is not perfect. I'm guessing the ranking features are based on the repo stats, contributor stats, etc. Even "good" contributors will make rookie mistakes in certain areas. Interesting to imagine how GH will try to solve this issue.

jpalomaki 5y ago
It might be possible to learn from the change history of the projects. There's likely quite many commits which fix certain security issues, such as SQL injection problems. Maybe even with suitable metadata in the issues or commit messages.

lmilcin 5y ago

I thought this should have been expected. Security starts with deep understanding. Some standards and practices can help avoid some types of problems, and some are even rather effective (like airgapping your systems), but there isn't any way to assure security in general other than truly understand what you are doing. ** I feel like Copilot is the wrong direction to optimize development. This is mostly going to help people with already poor understanding of what they are doing create even more crap. For a good developer those low level, low engagement activities are not a problem (except maybe for learning stage where you actually want people engaged rather than copy/paste). What it does not help is the important parts of development -- defining domain of your problem, design good APIs and abstractions, understanding how everything works and fits together, understanding what your client needs, etc. Also, I feel this is going to help increase complexity by making more copies of same structures throughout the codebase. My working theory about this is this is going to hinder new developers even more than they already are by google and stack*. Every time you are giving new developers an easier way to copy paste code without understanding you are robbing them an opportunity to gain deeper understanding of what they are doing and in effect prevent them from learning and growing. It is a little bit like giving answers to your kids homework without giving them chance to arrive at the answer or explaining anything about it. ** Another way I feel this is going to hurt developers is competition in who can produce most volume of code. I have already noticed this trend where developers (especially more junior but aspiring to advance) try to outcompete others by producing more code, close more tickets, etc. Right now it means skipping understanding of what is going on in favor of getting easy answers from the Internet. These guys can produce huge amounts of code with relatively little actual engagement. To management (especially with wrong incentives) this seems like a perfect worker, because management usually doesn't understand the connection between lack of engagement and planning at design/development time with their later problems (or they don't feel it is them that is going to pay the price). The Copilot is probably going to make it even more difficult for people who want to do it the right way because even starker difference in false productivity measurements.

bsenftner 5y ago
I too feel this is the wrong direction, from the fundamental aspect that trained algorithms contain no comprehension of what they are doing. They are the classic idiot savant. In a technological economy, comprehension of the environment is everything. I do not see how comprehension can be achieved without the elusive General AI, so I do not see this as anything other than a new area of research exposing how vitally important it is to have comprehension.
andix 5y ago
The real evil here is boilerplate code. I've seen so much boilerplate in the Java or classic .NET Framework world, it's incredible. So many layers of DTOs, Request/Response Models and so on, that could be just generated. Or most of the time even removed completely (that would cost some "architects" their job though). This is also true for a lot of Redux or Angular/NgRx applications. So much boilerplate, that you can't find the relevant code anymore.
- fn1 5y ago
  This is because "Copy'n'paste" programming get's more and more common. I see more and more juniors pasting code or shellcommands from StackOverflow with careless ease, without even pretending anymore that they're interested in how it actually works.
- lmilcin 5y ago
  (I have been professionally programming Java backends for the past 16 years). Java is not the culprit here. I think it is something that happened on the way that has something to do with J2EE and patterns craze we had a decade ago or two ago. It doesn't help that frameworks like Spring and their documentation go out of their way to propagate these boilerplate-heavy patters. Copying these lazy patterns is shortest, easiest way to get to working solution for a person that doesn't want to put any extra effort. And you can't get punished for doing this. Most developers don't even know there exist any other possibilities than mandatory controller calling service calling database layer and hordes of DTOs some people call "model".

wccrawford 5y ago

I'm actually impressed with that. There's so much insecure code out there that I'd have expected it to generate insecure code most of the time. I'd still not use it. But it's an impressive trick.

mbrevda1 5y ago

For comparison, what percentage of human-generated code is secure?

westurner 5y ago
> For comparison, what percentage of human-generated code is secure? Yeah how did they measure? Did static and dynamic analysis find design bugs too? Maybe - as part of a Copilot-assisted DevSecOps workflow involving static and dynamic analysis run by GitHub Actions CI - create Issues with CWE "Common Weakness Enumeration" URLs from e.g. the CWE Top 25 in order to train the team, and Pull Requests to fix each issue?: https://cwe.mitre.org/top25/ https://cwe.mitre.org/top25/ Which bots send PRs?
iainmerrick 5y ago
It seems reasonable to want Copilot to help you produce code of a reasonable quality. If it’s just helping you crank out the same bad code more quickly, without learning anything in the process, that’s useful to know. Some people might still want a tool like that, I wouldn’t.
- burnished 5y ago
  Sure. But in order to know if its 'of reasonable quality' you need some sort of baseline to compare it to. What is reasonable quality? I think what your average human does is probably reasonable. Like, if your average dev will produce insecure code in 80% of samples, then Copilot starts to look really good! But if its closer to 0.01% of code samples, then copilot looks more like an intriguing novelty, not to be brought too near serious work. Much like dippin dots in this regard.
eddieroger 5y ago
That's basically where my gut went when I read the headline - so is that of a junior engineer, or really any engineer who hasn't had to think about it, and we don't promote their code directly to prod, either (if we can avoid it). Copilot shouldn't be able to generate code destined for prod without review any more than should any line of code written by a human.

luke2m 5y ago

So what? Copilot isn't intended to replace humans, just help them out and maybe reduce typing.

toastal 5y ago

You are the free labor copilot to train Microsoft GitHub's Copilot tool. You are responsible for any of those insecure code errors and the diligence require. You will be on the hook for resulting problems. But Microsoft and their home-phoning, tracking-embedded editor will get real people to correct and train their machine for free—with their stated plan of later selling that machine back to us later. I wish there were a “robots.txt” file for Git to disallow certain bots from training on anything I have written.

gfiorav 5y ago
Plus, GH doesn't care how you licensed your code. It will learn from it and produced licensed code.
- deleted 5y ago
  [deleted]
carl_dr 5y ago
> I wish there were a “robots.txt” file for Git to disallow certain bots from training on anything I have written. It’s simple. If you are concerned by this, don’t host your repositories on GitHub.
- tyingq 5y ago
  You would have to not host your code publicly either, right?
  - nextlevelwizard 5y ago
    You can always host is with license that doesn't allow reuse or something
    - tyingq 5y ago
      GitHub mentions that they don't currently look at the license before trawling code. https://twitter.com/NoraDotCodes/status/1412741339771461635 https://twitter.com/NoraDotCodes/status/1412741339771461635 There's also other references that GitHub public repos weren't the only source. They trawl other publicly readable code.

evolveyourmind 5y ago

Meaning 40% of the code on GitHub is insecure

auggierose 5y ago
No. It means that when c = f(a, b), where a, and b are secure, and you have no clue what f does, it might still be the case that c is insecure.
dkersten 5y ago
You could train a model on purely secure code and still have it combine it in insecure ways.
- 0-_-0 5y ago
  And the other way around.
  - SketchySeaBeast 5y ago
    I guess the question really is how many ways are there to do it wrong, and how many ways are there to do it right?

moretti 5y ago

I use Copilot mostly as replacement for intellisense and macros. It helps me automating repetitive tasks. I would never trust Copilot for an algorithm or a snippet, I mean I would treat the code just like anything taken from StackOverflow or Github.

lvl100 5y ago

Why would anyone use this in production? Just use Sourcegraph if you need help that badly.

mullikine 5y ago

But this problem is solved using GitHub's CodeQL for searching and filtering generated code. By combining Copilot with GitHub Semantic and GitHub CodeQL, you have a means of writing and generating the code you want in a secure way. This means that you no longer need the original source code that was used to train Codex. Training Codex and selling as a product in the form of Copilot steals the essence of the original source code used to train it, to build the future of programming, while paying nothing back to the original authors. Even Elon Musk was opposed to OpenAI exclusively licensing to Microsoft GPT. https://edition.cnn.com/2020/09/27/tech/elon-musk-tesla-bill-gates-microsoft-open-ai/index.html https://edition.cnn.com/2020/09/27/tech/elon-musk-tesla-bill... It's so transformative that people may allow it to circumvent licenses.

gnrlst 5y ago

I've experienced this first hand: the autosuggest is scarily accurate and insidious at the same time. On numerous occasions I've auto-filled a 10-15 line suggestion that looked like it was exactly what I wanted to do, but made a very critical mistake (e.g. in a For loop, referencing the wrong array despite calling it the right name). Not really security related stuff, but head scratchers that make it harder to debug since I didn't actually write the code.

makach 5y ago

..does that means we will be 60% more secure than before?

dkersten 5y ago
Only if 100% of the code was insecure when written by a human. Given that humans can think through what code is doing, I don’t think that’s a reasonable assumption.

monkeydust 5y ago

I think where OpenAI Codex (which is what Copilot uses) gets more interesting is when the allow you to fine-tune the model on your own (trusted) code. That could help reduce the time it takes for new engineers to get up to speed for example.

bottled_poe 5y ago

What’s the baseline? 60% may still be superior to the average implementation.

lampe3 5y ago

I'm using copilot now for some time and yeah it's more a toy than real help right now. The only time it really helped when I needed to create a named list of char codes. When it comes to more complex code than checking the code of copilot takes the same time as writing it. 90% of the time I needed to correct copilot. For me, tools like linters are way more helpful then. If I could only use ESLint or copilot, I would go 100% of the time with ESLint.

harlekein 5y ago
I think another risk with getting Copilot to start out, is that it might nudge you into a direction you wouldn't have gone into otherwise. Whether that is better or not, I suppose, it depends.
- lampe3 5y ago
  I'm working on a HTML Tokenizer in Deno/Typescript. Copilot only helps with boilerplate code which could be handled by good intellisense. When it tries to generate a function from the function name it fails so hard that it is more in your way then helpful.

0-_-0 5y ago

I fail to see how this is particularly useful information about Copilot. The comparison should be: 1. How many times do people write insecure code when not using Copilot? 2. How many times do people write insecure code when using Copilot?

nextlevelwizard 5y ago
It is useful since it means copilot is not taking your job any time soon. i.e. if 40% of the time the human driving the thing is needed to intervene and prevent obvious security flaws then expert is still needed to use the tool.
- 0-_-0 5y ago
  I think it was obvious from the beginning that it's trained on GitHub code, so it would be surprising if it was better than the average code on GitHub. In any case, if Copilot can generate code as well as the average programmer without supervision, that means it can already take the job of 50% of programmers. A more useful metric though is how many programmers can a person using Copilot replace by having greater productivity? Also, in how many programming jobs does security matter? In my job for example it doesn't matter at all.
  - nextlevelwizard 5y ago
    With any commercial software security matters. You don't want your licensing solution to be craked by editing a file. Anything that connects to any server or has any sort of networking. Anything that needs privileges (like installing drivers) needs security. Anything that reads/interpeds any data given to it needs to have security. Of course this all depends what you mean by "matter". I can't come up with a program that doesn't need to think about security at all expect something trivial like hello world.

eurasiantiger 5y ago

I wonder if the Copilot model could somehow be repurposed to analyze the quality of a developer’s code. Seeing how Microsoft owns both GitHub and LinkedIn, it’s a good bet this is something they’re actively researching.

shireboy 5y ago

…Compared to 60% of circumstances in the meat-based developer control group? :)

spywaregorilla 5y ago
and of the population that is likely to use copilot in production for their own work? 90%?
- lupire 5y ago
  These are made up numbers. A control group is needed.
dimitrios1 5y ago
I love that we always use the average here for these justifications. We just slowly chip away and any and all excellence. 10x memes aside, we all know what it's like to work with a truly talented and productive engineer versus your everyday schmoe collecting a paycheck. It's a story as old as time, and yet here we are doing the exact big factory industrialization techniques other industries have done and that is commoditize the thing that made them exceptional and eliminating artisanship, uniqueness, and ultimately quality and character. It's a tragedy of the commons of a sort.
- phreeza 5y ago
  It may be a tragedy, but I fail to see why it is a tragedy of the commons? Which resource that is a available to all is being overused? High-paying dev jobs? Those are not a commons in the sense that tragedy of the commons implies because lower-quality devs don't stand to benefit by only taking a smaller part of the job.
  - runnerup 5y ago
    This actually feels to me like a concept worth exploring. I think we lack a concise term or phrase to reference what GP was trying to communicate. In my heart I feel similarly to GP - and it does feel a lot like how I feel about tragedy of the commons situations. Maybe there seems to be a shared opportunity for everyone if these private companies would make the most of their financial capital, market dominance, dominance in human resources, and most especially leverage their network effects. That would lead to better things for everyone, like the invention of smartphones. But the same corporations can also waste unimaginable resources and achieve very little. Often their failures don't just have little effect, but rather the failures choke/smother the market and prevent better alternatives from being widely used.

wcarss 5y ago

I couldn't find a link to the actual study anywhere in the article: https://arxiv.org/abs/2108.09293 https://arxiv.org/abs/2108.09293

lupire 5y ago

The thing about copilot that drives me crazy is that it is anti-good-programming. A smart AI should recommend a library function to call, not generate boilerplate code. The boilerplate code is either wrong, or should be a library function, nearly axiomatically.

arvindamirtaa 5y ago

Unit tests with TONS of assertions, cleaned data from form to ORM object, stuff that look look like you're just through a list and doing the same thing over and over. For these, Copilot is great. I wouldn't trust it to do anything else though. Nothing more. Nothing less.

softwaredoug 5y ago

I will say I’m not looking forward to writing some mundane code today. It’s interacting with GCS to scan a bucket for an extension, load the data with pandas, and concat some dataframes. It’s something dumb but mildly finicky that’s going to eat up so much time I could be using for higher value work. Copilot would be very welcome as I do this, instead of annoyingly going off to Google 3 different python libraries and getting it all to work nicely together.

ransom1538 5y ago

Has anyone here got past the wait list? I and my team members have been waiting for months.

queuebert 5y ago

This is exactly what an AGI would do if it wanted to pwn all our systems.

Vaslo 5y ago

It's learning from existing code, right? Doesn't this say something about developers in general, or is the thought that it uses combinations of code that are insecure?

harlekein 5y ago
I don't hold the average developer in very high regard. There are tons of developers who are much better than me and I readily read their books, follow their tweets, blog posts and online talks to learn from them. I hold them in high regard, but these people are not the average developer. If you would pick any smaller company with a dev team, a freelancer or an agency, your chances of finding a developer who understands and upholds quality code is vastly reduced. Not to mention a lot of beginners will just push their practice projects to GitHub and never look at it again. I'm also guilty of this, but I never realized Microsoft was training AI with this code. If Copilot is learning from these projects then I'd say the code it regurgitates is not average, but even below average.

cblconfederate 5y ago

Assuming that this is what it learned from its human counterparts, i'm surprised it 's so low.

rcarmo 5y ago

As many people have pointed out indirectly, this is almost certainly caused by the training set. Without a bias or ranking for quality, it will just churn out the “best fit” or most popular snippets…

amw-zero 5y ago

If it's trained on code that we write, that sounds completely accurate.

whazor 5y ago

Happily having access to GitHub copilot, it very often generates the code that I want. So it saves me from typing and also often saves checking Stack Overflow. I think the libraries/packages you use also play a big influence in how easy it is for copilot to create security flaws. Still, more training against security holes would be appreciated.

mzs 5y ago

the actual paper: https://arxiv.org/abs/2108.09293 https://arxiv.org/abs/2108.09293 previous discussion including comments from lead author: https://news.ycombinator.com/item?id=28279365 https://news.ycombinator.com/item?id=28279365

adamsvystun 5y ago

It is important to remember that Copilot can improve. 40% is not a bad baseline, but one data point does not give us much info, we should wait and see the rate of improvement.

dexen 5y ago

Half joking: so far GitHub Copilot is more feasible as tool for humans doing code-coverage for its input code, "given enough eyeballs, all bugs are shallow" style. When a developer goes, "huh, Copilot generated insecure code, better report it to the original project it learned it from" - if only Copilot was able to link to the original project, it would all be great and useful.

Animats 5y ago

Well, of course. GPT-3 has no underlying model of meaning. It's just autocomplete with a bigger data set. Used on natural language, it produces text that looks reasonable for about three paragraphs. Then you realize it's just blithering and has nothing to communicate. (Like too many bloggers, but that's another issue.)

cannabis_sam 5y ago

Of course it did, why would github copilot “care about” security, unless the majority of code on github cared about security?

COMMENT___ 5y ago

It's painful to see that GitHub Copilot is called "AI". For god's sake, it is not AI. It's just an advanced auto-complete for coders. GPT-3 is close to AI, GitHub Copilot is not. Jesus Christ, please make them stop. Stop using AI as a buzzword.

sprafa 5y ago
Is it fair to call it an ML based system?
- COMMENT___ 5y ago
  Why not? GitHub Copilot was trained with ML to autocomplete your code. But it is not AI.
arthur2e5 5y ago
GitHub Copilot is literally GPT adapted for code. The paper on OpenAI Codex, the stuff powering Copilot, makes it very clear in the abstract. https://arxiv.org/abs/2107.03374 https://arxiv.org/abs/2107.03374 Either you call both AI or you call neither AI. (A previous version of the comment stated that it was tuned from GPT-3. This is incorrect; the simpler GPT was used for faster convergence.)
- COMMENT___ 5y ago
  If I have to choose, then I would decide not to call them AI at all.

IlliOnato 5y ago

My take on Copilot has not changed. I believe it will make programmers that produce junk code more productive, by being able to produce more junk code in less time.

RhysU 5y ago

See also... You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completion https://deepai.org/publication/you-autocomplete-me-poisoning-vulnerabilities-in-neural-code-completion https://deepai.org/publication/you-autocomplete-me-poisoning...

spyder 5y ago

Well, it wasn't trained to output secure code was it?