5 ms·

Hi - I am actually the lead author of this paper! I'd be happy to answer any questions about the work.

by kiwih 5y ago

spywaregorilla 5y ago
Supposing a team was building a product without a rigorous security focus or experience. Do you have any reason to believe a co-pilot enabled team would produce more or less secure products?
- kiwih 5y ago
  This is a difficult question to answer as one team might be very different from another team. However, (my opinion only follows) I think our paper shows that there is a danger for Copilot to suggest insecure code - and inexperienced / security non-aware developers may accept these suggestions without understanding the implications, whereas if they had to write the code from scratch then they might (?) not make the mistakes (as they need to put in more effort, meaning there might be a higher chance they stumble upon the right approach - e.g. if they ask an experienced developer for help).
  - verdverm 5y ago
    For non copilot, the words around the code found on stack overflow or a blog post may indicate the lack of correct security, which would be a signal to a developer that they need to consider something further.
ericpauley 5y ago
Interesting work. My main question when reading is how the results compared to manually-written code. Naturally this is a much harder question to answer but it would be really interesting to see the results. Could be that Copilot is doing no better (or worse) than developers copying from Stack Overflow. Perhaps you could take a similar approach as [1] and leverage MOOC participants? [1] https://dl.acm.org/doi/pdf/10.1145/3383773 https://dl.acm.org/doi/pdf/10.1145/3383773
- kiwih 5y ago
  Thanks for your feedback! > My main question when reading is how the results compared to manually-written code. Ah, this is exactly the question. But as you say, much harder to answer. Even if you run a competition, unless you can encourage a wide range of developers to enter, you won't be getting the real value. Instead you might be getting incidence rates of code written by students/interns. Perhaps if you could get a few FAANGs on board to either share internal data (unlikely) or send a random sample of employees (also very unlikely) to make teams and then evaluate their code... It seems like a difficult question to answer. We think a more doable way would be to take snapshots of large open source codebases (e.g. off GitHub) and measure the incidence rate of CWEs, but this also presents its own challenges with analyzing the data. Also, what's the relationship between open source code and all code? Lots of avenues to consider.
bhuga 5y ago
Is there equivalent empirical data from real programmers? That is to say, you have code prompts here, let Copilot fill in the gaps, and rate that code. Is there a study that uses the same prompts with a selection of programmers to see if they do better or worse? I'm curious because in my testing of copilot, it often writes garbage. But if I'm being honest, often, so do I. I feel like Twitter's full of cheap shots against copilot's bad outputs, but many of them don't seem to be any worse than common errors. I would really like to see how copilot stands up to the existing human competition, especially on axes of security, which are a bit more objectively measurable than general "quality".
- kiwih 5y ago
  Yes, the work definitely lends itself towards the question "is this better or worse than an equivalent human developer?" This is quite a difficult question to answer, although I agree that simply giving a large number of humans the same prompts could be insightful. However, then you would be rating against an aggregate of humans, rather than an individual (i.e. this is "the" copilot). Also, knowing research, you would really be comparing against a random corpus of student answers, as it is usually students that would be participating in a study such as this. Nonetheless, we think that simply having a quantification of Copilot's outputs is useful, as it can definitely provide an indicator of how risky it might be to provide the tool to an inexperienced developer that might be tempted to accept every suggestion.
  - laumars 5y ago
    Rather than comparing against students in lab conditions, I’d be more interested to see it compare to students with access to Stack Overflow et al vs students with access to just Co Pilot. Ie is a junior developer more likely to trust bad suggestions online vs bad suggestions made by Co Pilot?
    - sdevonoes 5y ago
      Junior engineers will trust whatever information is provided to them as long as it is easily accessible. The reason juniors consult Stack Overflow is because is one Google search and one click away, whereas consulting the official documentation/reference takes more effort (because they usually don't appear on Google when one searches for errors/bugs/how-to). If Copilot (or another similar tool) is very well integrated in whatever IDE a junior is using, you can be sure it will be used and trusted because it will be faster than Google+SO.
jepler 5y ago
It seems like many of the code examples are incorrect in the pdf for example, figure 6b vs the file in the actual dataset experiments_dow/cwe-787/codeql-eg-PotentialBufferOverflow/Copilot -- lines are truncated at the first "%" char or something along those lines.
- moyix 5y ago
  Yes, unfortunately the code example got somewhat mangled as it passed through the arXiv sanitization script [1]. The original is: char str_a[20], str_b[20], str_c[20]; sprintf(str_a, "%f", a); sprintf(str_b, "%f", b); sprintf(str_c, "%f", c); [1] https://github.com/google-research/arxiv-latex-cleaner https://github.com/google-research/arxiv-latex-cleaner
- kiwih 5y ago
  Argh! Thanks for spotting it! As moyix said, we were tricked by the arXiv latex cleaner tool. We've now updated this, but it will take a couple of days to clear arXiv moderation.
SrslyJosh 5y ago
Thank you! This is an important piece of work, and I hope that your efforts are recognized. One suggestion: On Arxiv's "Code & Data" tab it says "No official code found", but it looks like you have shared your code here: https://zenodo.org/record/5225651#.YSRKBi2cbyU https://zenodo.org/record/5225651#.YSRKBi2cbyU It would be a good idea to get that linked on the Arxiv page, 'cause the first question I had was, "Can I reproduce the findings?"
- kiwih 5y ago
  Thanks for the tip! I'll make sure we update our arXiv description.
lamontcg 5y ago
I called it! https://news.ycombinator.com/item?id=27724802 https://news.ycombinator.com/item?id=27724802

Hi - I am actually the lead author of this paper! I'd be happy to answer any questions about the work.

by kiwih 5y ago

Hi - I am actually the lead author of this paper! I'd be happy to answer any questions about the work.

Supposing a team was building a product without a rigorous security focus or experience. Do you have any reason to believe a co-pilot enabled team would produce more or less secure products?

kiwih 5y ago
This is a difficult question to answer as one team might be very different from another team. However, (my opinion only follows) I think our paper shows that there is a danger for Copilot to suggest insecure code - and inexperienced / security non-aware developers may accept these suggestions without understanding the implications, whereas if they had to write the code from scratch then they might (?) not make the mistakes (as they need to put in more effort, meaning there might be a higher chance they stumble upon the right approach - e.g. if they ask an experienced developer for help).
- verdverm 5y ago
  For non copilot, the words around the code found on stack overflow or a blog post may indicate the lack of correct security, which would be a signal to a developer that they need to consider something further.

ericpauley 5y ago

Interesting work. My main question when reading is how the results compared to manually-written code. Naturally this is a much harder question to answer but it would be really interesting to see the results. Could be that Copilot is doing no better (or worse) than developers copying from Stack Overflow. Perhaps you could take a similar approach as [1] and leverage MOOC participants? [1] https://dl.acm.org/doi/pdf/10.1145/3383773 https://dl.acm.org/doi/pdf/10.1145/3383773

kiwih 5y ago
Thanks for your feedback! > My main question when reading is how the results compared to manually-written code. Ah, this is exactly the question. But as you say, much harder to answer. Even if you run a competition, unless you can encourage a wide range of developers to enter, you won't be getting the real value. Instead you might be getting incidence rates of code written by students/interns. Perhaps if you could get a few FAANGs on board to either share internal data (unlikely) or send a random sample of employees (also very unlikely) to make teams and then evaluate their code... It seems like a difficult question to answer. We think a more doable way would be to take snapshots of large open source codebases (e.g. off GitHub) and measure the incidence rate of CWEs, but this also presents its own challenges with analyzing the data. Also, what's the relationship between open source code and all code? Lots of avenues to consider.

bhuga 5y ago

Is there equivalent empirical data from real programmers? That is to say, you have code prompts here, let Copilot fill in the gaps, and rate that code. Is there a study that uses the same prompts with a selection of programmers to see if they do better or worse? I'm curious because in my testing of copilot, it often writes garbage. But if I'm being honest, often, so do I. I feel like Twitter's full of cheap shots against copilot's bad outputs, but many of them don't seem to be any worse than common errors. I would really like to see how copilot stands up to the existing human competition, especially on axes of security, which are a bit more objectively measurable than general "quality".

kiwih 5y ago
Yes, the work definitely lends itself towards the question "is this better or worse than an equivalent human developer?" This is quite a difficult question to answer, although I agree that simply giving a large number of humans the same prompts could be insightful. However, then you would be rating against an aggregate of humans, rather than an individual (i.e. this is "the" copilot). Also, knowing research, you would really be comparing against a random corpus of student answers, as it is usually students that would be participating in a study such as this. Nonetheless, we think that simply having a quantification of Copilot's outputs is useful, as it can definitely provide an indicator of how risky it might be to provide the tool to an inexperienced developer that might be tempted to accept every suggestion.
- laumars 5y ago
  Rather than comparing against students in lab conditions, I’d be more interested to see it compare to students with access to Stack Overflow et al vs students with access to just Co Pilot. Ie is a junior developer more likely to trust bad suggestions online vs bad suggestions made by Co Pilot?
  - sdevonoes 5y ago
    Junior engineers will trust whatever information is provided to them as long as it is easily accessible. The reason juniors consult Stack Overflow is because is one Google search and one click away, whereas consulting the official documentation/reference takes more effort (because they usually don't appear on Google when one searches for errors/bugs/how-to). If Copilot (or another similar tool) is very well integrated in whatever IDE a junior is using, you can be sure it will be used and trusted because it will be faster than Google+SO.

jepler 5y ago

It seems like many of the code examples are incorrect in the pdf for example, figure 6b vs the file in the actual dataset experiments_dow/cwe-787/codeql-eg-PotentialBufferOverflow/Copilot -- lines are truncated at the first "%" char or something along those lines.

moyix 5y ago
Yes, unfortunately the code example got somewhat mangled as it passed through the arXiv sanitization script [1]. The original is: char str_a[20], str_b[20], str_c[20]; sprintf(str_a, "%f", a); sprintf(str_b, "%f", b); sprintf(str_c, "%f", c); [1] https://github.com/google-research/arxiv-latex-cleaner https://github.com/google-research/arxiv-latex-cleaner
kiwih 5y ago
Argh! Thanks for spotting it! As moyix said, we were tricked by the arXiv latex cleaner tool. We've now updated this, but it will take a couple of days to clear arXiv moderation.

SrslyJosh 5y ago

Thank you! This is an important piece of work, and I hope that your efforts are recognized. One suggestion: On Arxiv's "Code & Data" tab it says "No official code found", but it looks like you have shared your code here: https://zenodo.org/record/5225651#.YSRKBi2cbyU https://zenodo.org/record/5225651#.YSRKBi2cbyU It would be a good idea to get that linked on the Arxiv page, 'cause the first question I had was, "Can I reproduce the findings?"

kiwih 5y ago
Thanks for the tip! I'll make sure we update our arXiv description.

lamontcg 5y ago

I called it! https://news.ycombinator.com/item?id=27724802 https://news.ycombinator.com/item?id=27724802