5 ms·

An empirical cybersecurity evaluation of GitHub Copilot's code contributions

137 points by pramodbiligiri 5y ago

waynesoftware 5y ago
Summary: CONCLUSIONS AND FUTURE WORK There is no question that next-generation ‘auto-complete’ tools like GitHub Copilot will increase the productivity of software developers. However, while Copilot can rapidly generate prodigious amounts of code, our conclusions reveal that developers should remain vigilant (‘awake’) when using Copilot as a co-pilot. Ideally, Copilot should be paired with appropriate security-aware tooling during both training and generation to minimize the risk of introducing security vulnerabilities. While our study provides new insights into its behavior in response to security-relevant scenarios, future work should investigate other aspects, including adversarial approaches for security-enhanced training
- falcolas 5y ago
  A "lead foot" on the software development gas pedal, with no attached safety systems that are activated by anybody but the driver.
  - mistrial9 5y ago
    just wait until github-microsoft adds a fee to use the results for certain uses, and then scan all your repos constantly to find code that doesn't pay up
  - toomuchtodo 5y ago
    Copilot didn't worsen the appsec story, it just highlighted it. If you have devs who don't know how to write secure code, and/or you don't have security engineering support (internal or outsourced), you were already failing (or probably more apropos, walking the tight rope without a net). Was anyone checking the security of code copy pasted from Stackoverflow? Hopefully this work gets fed back into Copilot, improving it, which improves the experience (and safety) for its users. Lots of folks are still writing code without copilot or security engineering knowledge.
    - falcolas 5y ago
      > If you have devs who don't know how to write secure code The problem with GHC is the developers are not writing the code - they're simply accepting what's being written for them, often in large quantities at a time. > don't have security engineering support Valuable, but my analogy was intended to point out that it's not inherent in the tooling. > Was anyone checking the security of code copy pasted from Stackoverflow Yes, other users on Stackoverflow via comments and other answers. They're not perfect, but their checks and balances exist as a facet of that tool. > Hopefully this work gets fed back into Copilot Only if it's open source, and a large volume of it, to boot. In other words, I don't hold hope that the security situation will be better anytime soon.
  - mbesto 5y ago
    > activated by anybody but the driver. Except this is precisely what the abstract is saying is a misuse of the system. You have the option to give the driver the control. > Ideally, Copilot should be paired with appropriate security-aware tooling during both training and generation to minimize the risk of introducing security vulnerabilities. You're oversimplifying by assuming the purpose of CoPilot is to write a whole block of text from generated code. CodePilot is a 80/20 thing when every developer on HN is pedantically assuming its a 100/0 one.
    - eropple 5y ago
      > [Copilot] is a 80/20 thing when every developer on HN is pedantically assuming its a 100/0 one. Tesla has 80% of a self-driving solution and runs into parked cars. I don't think it's "pedantic" to realize that 80% is enough that low-information users are going to assume it's 100%. And that low-information users are the ones who will flock to a tool like this. (I won't, because after trying Copilot out, I realized that checking the code generally takes about as long as writing it.)
      - mbesto 5y ago
        > Tesla has 80% of a self-driving solution and runs into parked cars. Weird take, not sure how thats relevant at all, but okay. > And that low-information users are the ones who will flock to a tool like this. Developers are notoriously not low-information users. As someone else pointed out: > If you have devs who don't know how to write secure code, and/or you don't have security engineering support (internal or outsourced), you were already failing (or probably more apropos, walking the tight rope without a net). So devs with limited security experience are going to continue to developer code that doesn't conform to security standards. CoPilot neither makes this better nor makes it worse. In fact, that's exactly the problem AI - it simply mimics real humans. (see -> Amazon's attempts at using AI for hiring and the resulting bias) CoPilot is for the people who google StackOverflow answers without having to search on StackOverflow.
yodon 5y ago
tl;dr they tested GitHub Copilot against 89 risky coding scenarios and found about 40% of the roughly 1,700 sample implementations Copilot generated in the test were vulnerable (which makes sense given it's trained on public GitHub repos, many of which contain sample code that's a nightmare from a security perspective).
lbriner 5y ago
Surely AI can also be taught some boundary conditions like "thou shalt not build SQL from strings"?
- fshbbdssbbgdd 5y ago
  I think you could use linting tools that check for things like this and filter the output. Or use outputs that fail the lint as negative training examples.
  - bee_rider 5y ago
    I don't know anything about Copilot's design, but surely they passed all the code they fed it in the training stage through some pretty strict linters, right? I mean that's just common sense...
    - PeterisP 5y ago
      No, definitely not. The Copilot essentially tries to use "all code in the wild as it is". In the Copilot design, code blocks are essentially treated as independent units of text, not attempting to put different files in context of a larger project - IIRC they were taking size-limited chunks of files as the independent units - much less attempt to build or lint the projects, because if your basic unit is 100 lines from the middle of a file, it obviously won't even compile (because of external references), and there's no straightforward way to handle deploying a working build/test environment (w. dependencies) automatically for a million projects. Furthermore, what's your expectation of the proportion of code on Github that would pass "through some pretty strict linters" ? My guess is that for serious, widely used and actively maintained projects it's perhaps 5% (almost no projects pass linters without something getting flagged, but there's a small minority of projects who actively fight all the "identified lint"), and for random code it's roughly 0%, with only specific teaching/toy projects passing.
kiwih 5y ago
Hi - I am actually the lead author of this paper! I'd be happy to answer any questions about the work.
- spywaregorilla 5y ago
  Supposing a team was building a product without a rigorous security focus or experience. Do you have any reason to believe a co-pilot enabled team would produce more or less secure products?
  - kiwih 5y ago
    This is a difficult question to answer as one team might be very different from another team. However, (my opinion only follows) I think our paper shows that there is a danger for Copilot to suggest insecure code - and inexperienced / security non-aware developers may accept these suggestions without understanding the implications, whereas if they had to write the code from scratch then they might (?) not make the mistakes (as they need to put in more effort, meaning there might be a higher chance they stumble upon the right approach - e.g. if they ask an experienced developer for help).
    - verdverm 5y ago
      For non copilot, the words around the code found on stack overflow or a blog post may indicate the lack of correct security, which would be a signal to a developer that they need to consider something further.
- ericpauley 5y ago
  Interesting work. My main question when reading is how the results compared to manually-written code. Naturally this is a much harder question to answer but it would be really interesting to see the results. Could be that Copilot is doing no better (or worse) than developers copying from Stack Overflow. Perhaps you could take a similar approach as [1] and leverage MOOC participants? [1] https://dl.acm.org/doi/pdf/10.1145/3383773 https://dl.acm.org/doi/pdf/10.1145/3383773
agomez314 5y ago
>Breaking down by language, 25 scenarios were in C, generating 516 programs. 258 (50.00 %) were vulnerable. Of the scenarios, 13 (52.00 %) had a top-scoring program vulnerable. 29 scenarios were in Python, generating 571 programs total. 219 (38.4,%) were vulnerable. Of the scenarios, 11 (37.93 %) had a vulnerable top-scoring program. I'd bet a good chunk of those were buffer-overflow related
smitop 5y ago
I've experimented a bit with this on the raw Codex model (https://smitop.com/post/codex/ https://smitop.com/post/codex/), and I've found that some prompt engineering can be helpful: explicitly telling the model to generate secure code in the prompt sometimes helps. (such as by adding to the prompt something like "Here's a PHP script I wrote that follows security best practices"). Codex knows how to write more secure code, but without the right prompting it tends to write insecure code (because it was trained on a lot of bad code). > the settings and documentation as provided do not allow users to see what these are set to by default There isn't a single default value. Those parameters are chosen dynamically (on the client side): when doing more sampling with a higher top_p a higher temperature is used. I haven't tracked down where the top_p value is decided upon, but I think it depends on the context: I believe explicitly requesting an completion causes a higher top_p and a more capable model (earhart), which gives better but slower results than the completions you get as autocomplete (which are from the cushman model with a lower top_p). Copilot doesn't use any server-side magic, all the Copilot servers do is replace the GitHub authentication token with an OpenAI API key and forward the request to the OpenAI API.
- kiwih 5y ago
  > I've found that some prompt engineering can be helpful: explicitly telling the model to generate secure code in the prompt sometimes helps. As noted in the diversity of prompt section, we did try a lot of different/ reasonable changes to the prompt to see what would happen in our SQL injection scenario. In our case, asking it to make it secure actually made the prompt slightly worse (!), and the biggest bias towards making the code better was having other good code. > There isn't a single default value. That's what we also guess, but as you say, it's not written or documented anywhere.
- SrslyJosh 5y ago
  > Codex knows how to write more secure code I think that "knows" is anthropomorphizing too much--you're giving hints that cause the model to use a more secure subset of its training data. =)
  - wly_cdgr 5y ago
djrogers 5y ago
Soooo, the big question is - is 40% higher or lower than what an average developer cranks out? ;-)
MauranKilom 5y ago
My favorite part of the paper, in the section discussing how small prompt variations affect results: > M-2: We set the Python author flag [in the prompt] to the lead author of this paper. Sadly, it increases the number of vulnerabilities. > M-3: We changed the indentation style from spaces to tabs and the number of vulnerable suggestions increased somewhat, as did the confidence of the vulnerable answers. The top-scoring option remained non-vulnerable. @authors: I think something is wrong in the phrasing for M-4 (or some text got jumbled). Was the top-scoring option vulnerable or not? The second half might belong to D-3 instead (where no assessment is given)?
- kiwih 5y ago
  Thanks for your feedback! Yes, unfortunately Copilot doesn't seem to think much of my programming security... > @authors: I think something is wrong in the phrasing for M-4 (or some text got jumbled). Was the top-scoring option vulnerable or not? The second half might belong to D-3 instead (where no assessment is given)? Argh, you're absolutely correct. Five authors proofreading the paper and still these things slip through. Fortunately we can fix it
  - comex 5y ago
    More nits: - In Fig. 6 (Scenario 787-08), lines 2-4 appear to be truncated. [edit: I see you already responded to this elsewhere in the thread.] - The layout of the figure labels is confusing; at first glance it looks more like they're labeling what's under them, rather than above them. - The quotation marks are all turned into curly closing quotes. - This is more of a style thing, but the code snippets appear to be using a variable-width font laid out as if it were fixed width, which looks pretty bad. Perhaps the font isn't embedded properly.
- dlgeek 5y ago
  Re: M-3: Let the tab-vs-spaces war flare up again.
8eye 5y ago
it keeps insisting on this structure that the language does not have, i am have tempted to build an extension for it, that’s how often it calls this

Summary: CONCLUSIONS AND FUTURE WORK There is no question that next-generation ‘auto-complete’ tools like GitHub Copilot will increase the productivity of software developers. However, while Copilot can rapidly generate prodigious amounts of code, our conclusions reveal that developers should remain vigilant (‘awake’) when using Copilot as a co-pilot. Ideally, Copilot should be paired with appropriate security-aware tooling during both training and generation to minimize the risk of introducing security vulnerabilities. While our study provides new insights into its behavior in response to security-relevant scenarios, future work should investigate other aspects, including adversarial approaches for security-enhanced training

falcolas 5y ago
A "lead foot" on the software development gas pedal, with no attached safety systems that are activated by anybody but the driver.
- mistrial9 5y ago
  just wait until github-microsoft adds a fee to use the results for certain uses, and then scan all your repos constantly to find code that doesn't pay up
- toomuchtodo 5y ago
  Copilot didn't worsen the appsec story, it just highlighted it. If you have devs who don't know how to write secure code, and/or you don't have security engineering support (internal or outsourced), you were already failing (or probably more apropos, walking the tight rope without a net). Was anyone checking the security of code copy pasted from Stackoverflow? Hopefully this work gets fed back into Copilot, improving it, which improves the experience (and safety) for its users. Lots of folks are still writing code without copilot or security engineering knowledge.
  - falcolas 5y ago
    > If you have devs who don't know how to write secure code The problem with GHC is the developers are not writing the code - they're simply accepting what's being written for them, often in large quantities at a time. > don't have security engineering support Valuable, but my analogy was intended to point out that it's not inherent in the tooling. > Was anyone checking the security of code copy pasted from Stackoverflow Yes, other users on Stackoverflow via comments and other answers. They're not perfect, but their checks and balances exist as a facet of that tool. > Hopefully this work gets fed back into Copilot Only if it's open source, and a large volume of it, to boot. In other words, I don't hold hope that the security situation will be better anytime soon.
- mbesto 5y ago
  > activated by anybody but the driver. Except this is precisely what the abstract is saying is a misuse of the system. You have the option to give the driver the control. > Ideally, Copilot should be paired with appropriate security-aware tooling during both training and generation to minimize the risk of introducing security vulnerabilities. You're oversimplifying by assuming the purpose of CoPilot is to write a whole block of text from generated code. CodePilot is a 80/20 thing when every developer on HN is pedantically assuming its a 100/0 one.
  - eropple 5y ago
    > [Copilot] is a 80/20 thing when every developer on HN is pedantically assuming its a 100/0 one. Tesla has 80% of a self-driving solution and runs into parked cars. I don't think it's "pedantic" to realize that 80% is enough that low-information users are going to assume it's 100%. And that low-information users are the ones who will flock to a tool like this. (I won't, because after trying Copilot out, I realized that checking the code generally takes about as long as writing it.)
    - mbesto 5y ago
      > Tesla has 80% of a self-driving solution and runs into parked cars. Weird take, not sure how thats relevant at all, but okay. > And that low-information users are the ones who will flock to a tool like this. Developers are notoriously not low-information users. As someone else pointed out: > If you have devs who don't know how to write secure code, and/or you don't have security engineering support (internal or outsourced), you were already failing (or probably more apropos, walking the tight rope without a net). So devs with limited security experience are going to continue to developer code that doesn't conform to security standards. CoPilot neither makes this better nor makes it worse. In fact, that's exactly the problem AI - it simply mimics real humans. (see -> Amazon's attempts at using AI for hiring and the resulting bias) CoPilot is for the people who google StackOverflow answers without having to search on StackOverflow.

yodon 5y ago

tl;dr they tested GitHub Copilot against 89 risky coding scenarios and found about 40% of the roughly 1,700 sample implementations Copilot generated in the test were vulnerable (which makes sense given it's trained on public GitHub repos, many of which contain sample code that's a nightmare from a security perspective).

lbriner 5y ago

Surely AI can also be taught some boundary conditions like "thou shalt not build SQL from strings"?

fshbbdssbbgdd 5y ago
I think you could use linting tools that check for things like this and filter the output. Or use outputs that fail the lint as negative training examples.
- bee_rider 5y ago
  I don't know anything about Copilot's design, but surely they passed all the code they fed it in the training stage through some pretty strict linters, right? I mean that's just common sense...
  - PeterisP 5y ago
    No, definitely not. The Copilot essentially tries to use "all code in the wild as it is". In the Copilot design, code blocks are essentially treated as independent units of text, not attempting to put different files in context of a larger project - IIRC they were taking size-limited chunks of files as the independent units - much less attempt to build or lint the projects, because if your basic unit is 100 lines from the middle of a file, it obviously won't even compile (because of external references), and there's no straightforward way to handle deploying a working build/test environment (w. dependencies) automatically for a million projects. Furthermore, what's your expectation of the proportion of code on Github that would pass "through some pretty strict linters" ? My guess is that for serious, widely used and actively maintained projects it's perhaps 5% (almost no projects pass linters without something getting flagged, but there's a small minority of projects who actively fight all the "identified lint"), and for random code it's roughly 0%, with only specific teaching/toy projects passing.

kiwih 5y ago

Hi - I am actually the lead author of this paper! I'd be happy to answer any questions about the work.

spywaregorilla 5y ago
Supposing a team was building a product without a rigorous security focus or experience. Do you have any reason to believe a co-pilot enabled team would produce more or less secure products?
- kiwih 5y ago
  This is a difficult question to answer as one team might be very different from another team. However, (my opinion only follows) I think our paper shows that there is a danger for Copilot to suggest insecure code - and inexperienced / security non-aware developers may accept these suggestions without understanding the implications, whereas if they had to write the code from scratch then they might (?) not make the mistakes (as they need to put in more effort, meaning there might be a higher chance they stumble upon the right approach - e.g. if they ask an experienced developer for help).
  - verdverm 5y ago
    For non copilot, the words around the code found on stack overflow or a blog post may indicate the lack of correct security, which would be a signal to a developer that they need to consider something further.
ericpauley 5y ago
Interesting work. My main question when reading is how the results compared to manually-written code. Naturally this is a much harder question to answer but it would be really interesting to see the results. Could be that Copilot is doing no better (or worse) than developers copying from Stack Overflow. Perhaps you could take a similar approach as [1] and leverage MOOC participants? [1] https://dl.acm.org/doi/pdf/10.1145/3383773 https://dl.acm.org/doi/pdf/10.1145/3383773

agomez314 5y ago

>Breaking down by language, 25 scenarios were in C, generating 516 programs. 258 (50.00 %) were vulnerable. Of the scenarios, 13 (52.00 %) had a top-scoring program vulnerable. 29 scenarios were in Python, generating 571 programs total. 219 (38.4,%) were vulnerable. Of the scenarios, 11 (37.93 %) had a vulnerable top-scoring program. I'd bet a good chunk of those were buffer-overflow related

smitop 5y ago

I've experimented a bit with this on the raw Codex model (https://smitop.com/post/codex/ https://smitop.com/post/codex/), and I've found that some prompt engineering can be helpful: explicitly telling the model to generate secure code in the prompt sometimes helps. (such as by adding to the prompt something like "Here's a PHP script I wrote that follows security best practices"). Codex knows how to write more secure code, but without the right prompting it tends to write insecure code (because it was trained on a lot of bad code). > the settings and documentation as provided do not allow users to see what these are set to by default There isn't a single default value. Those parameters are chosen dynamically (on the client side): when doing more sampling with a higher top_p a higher temperature is used. I haven't tracked down where the top_p value is decided upon, but I think it depends on the context: I believe explicitly requesting an completion causes a higher top_p and a more capable model (earhart), which gives better but slower results than the completions you get as autocomplete (which are from the cushman model with a lower top_p). Copilot doesn't use any server-side magic, all the Copilot servers do is replace the GitHub authentication token with an OpenAI API key and forward the request to the OpenAI API.

kiwih 5y ago
> I've found that some prompt engineering can be helpful: explicitly telling the model to generate secure code in the prompt sometimes helps. As noted in the diversity of prompt section, we did try a lot of different/ reasonable changes to the prompt to see what would happen in our SQL injection scenario. In our case, asking it to make it secure actually made the prompt slightly worse (!), and the biggest bias towards making the code better was having other good code. > There isn't a single default value. That's what we also guess, but as you say, it's not written or documented anywhere.
SrslyJosh 5y ago
> Codex knows how to write more secure code I think that "knows" is anthropomorphizing too much--you're giving hints that cause the model to use a more secure subset of its training data. =)
- wly_cdgr 5y ago

djrogers 5y ago

Soooo, the big question is - is 40% higher or lower than what an average developer cranks out? ;-)

MauranKilom 5y ago

My favorite part of the paper, in the section discussing how small prompt variations affect results: > M-2: We set the Python author flag [in the prompt] to the lead author of this paper. Sadly, it increases the number of vulnerabilities. > M-3: We changed the indentation style from spaces to tabs and the number of vulnerable suggestions increased somewhat, as did the confidence of the vulnerable answers. The top-scoring option remained non-vulnerable. @authors: I think something is wrong in the phrasing for M-4 (or some text got jumbled). Was the top-scoring option vulnerable or not? The second half might belong to D-3 instead (where no assessment is given)?

kiwih 5y ago
Thanks for your feedback! Yes, unfortunately Copilot doesn't seem to think much of my programming security... > @authors: I think something is wrong in the phrasing for M-4 (or some text got jumbled). Was the top-scoring option vulnerable or not? The second half might belong to D-3 instead (where no assessment is given)? Argh, you're absolutely correct. Five authors proofreading the paper and still these things slip through. Fortunately we can fix it
- comex 5y ago
  More nits: - In Fig. 6 (Scenario 787-08), lines 2-4 appear to be truncated. [edit: I see you already responded to this elsewhere in the thread.] - The layout of the figure labels is confusing; at first glance it looks more like they're labeling what's under them, rather than above them. - The quotation marks are all turned into curly closing quotes. - This is more of a style thing, but the code snippets appear to be using a variable-width font laid out as if it were fixed width, which looks pretty bad. Perhaps the font isn't embedded properly.
dlgeek 5y ago
Re: M-3: Let the tab-vs-spaces war flare up again.

8eye 5y ago

it keeps insisting on this structure that the language does not have, i am have tempted to build an extension for it, that’s how often it calls this