18 ms·
Security weaknesses of Copilot generated code in GitHub
- belter 3y ago"...The results show that (1) 35.8% of Copilot generated code snippets contain CWEs, and those issues are spread across multiple languages, (2) the security weaknesses are diverse and related to 42 different CWEs, in which CWE-78: OS Command Injection, CWE-330: Use of Insufficiently Random Values, and CWE-703: Improper Check or Handling of Exceptional Conditions occurred the most frequently, and (3) among the 42 CWEs identified, 11 of those belong to the currently recognized 2022 CWE Top-25. Our findings confirm that developers should be careful when adding code generated by Copilot (and similar AI code generation tools) and should also run appropriate security checks as they accept the suggested code..."
- jncfhnb 3y agoDid they prompt it to consider security weaknesses?
- OmarShehata 3y agoThat would be kind of wild. Imagine a world where whether your system was secure was just a matter of remembering to tell the AI agent "& also make it secure" before it writes your code. (could be quite real!)
- Grimburger 3y ago> also make it secure [proceeds to simply refactor the same code]
- jncfhnb 3y agoI think it’s more likely that you would use a security graded bot. It’s perfectly reasonable to not use secure code for a large number of use cases.
- phyzome 3y agoThis would likely help a little bit. We've already seen LLMs improve performance on some tasks by being instructed to "think carefully" first; presumably this biases it towards parts of the training set that are higher quality. But security ultimately requires comprehension, which is not something LLMs have.
- jncfhnb 3y agoSecurity 100% does not require comprehension in the philosophical sense
- phyzome 3y agoNot sure what the philosophical sense would be. I just mean that an awful lot of people treat security as "hack on the code until it doesn't obviously break", and that's the wrong mindset for practical security.
- jncfhnb 3y agoI’m not sure what you could possibly mean by LLMs don’t have comprehension if not speaking to philosphical debates of what comprehension means. I think your point is otherwise right, but the correct answer is standard best practices which is the easiest thing for bots to do
- deleted 3y ago[deleted]
- prosim 3y agoThey did not prompt at all. They used GitHub’s code search to find projects where the repo owner specified that the code was generated “by Copilot” and the authors took that at face value for all code in the project. Whether the code was actually suggested by Copilot is not at all analyzed in the paper. As such, the results are highly questionable.
- faeriechangling 3y agoIf a weakness is common, then of course Copilot is going to suggest it. Copilot gives you popular responses not correct ones. Yet if a weakness is common, it also means that human coders frequently make the same mistake as well. The studies results are rather unsurprising and its conclusions are oft-repeated advice. As many have said, treat copilot’s code in the same light you would treat a junior programmer’s code.
- pylua 3y agoI wonder if llm are biased towards older, more insecure implementations because there is a higher volume of old code vs new code. Same thing with the data it is trained on — not all code requires all levels of refinement. Most of the data is probably around average.
- squigz 3y agoI'm not sure that code being newer inherently means it will be more secure
- pylua 3y agoI don’t think it is a tautology , but I can imagine a cve scanner picking up older code with log4j where newer code may avoid that library altogether, just as an example. Since there is more older code than newer code would the llm be suspectible to that ?
- xmcqdpt2 3y agoEveryone still uses log4j after the fix. There aren't many full-featured alternatives... and the one that exist probably contain unfixed bugs.
- darkerside 3y ago> Most of the data is probably around average. I know this is not how distributions work, but I had to chuckle at the literal interpretation of this.
- calibas 3y ago> The results show that (1) 35.8% of Copilot generated code snippets contain CWEs What percent of non-Copilot generated public GitHub repos contain CWEs? Edit: According to this study, Copilot generates C/C++ code with vulnerabilities, but at a lower rate than your average human coder: https://arxiv.org/pdf/2204.04741.pdf https://arxiv.org/pdf/2204.04741.pdf
- gmerc 3y agoAs always the statistic is useless without the human comparison. If it improves on human coders, no amount of gnashing and wailing will stop the layoffs.
- ResearchCode 3y agoThey didn't improve on human truck drivers yet.
- laurent_du 3y agoI wonder if it would be possible to rate the code used during the training phase. For example the code could go through various static analysis tools and the result would be assigned as metadata to the code being used to train the model. The final model would then know that a given pattern is flagged as problematic by some tool and could take this into account not just to suggest new snippets but also to suggest improvements of existing snippets. Though I suppose if it was that easy, they'd have done it already.
- jasfi 3y agoThis is probably the next step for the LLM providers. They need to find ways to increase quality, and for code, there are many options. Perhaps code repos could get in on this too.
- progval 3y agoWouldn't this train it to avoid detection more than to avoid bad patterns?
- ivancho 3y agoYes, but presumably in the training data those two are quite correlated.
- tedunangst 3y agoThere's only one weakness specifically identified that I can see. print("new user", username, password) Yeah, not best practice, but also pretty common for development if you wanted to check that everything is being passed to the correct function.
- jddj 3y agoI don't know if it still does it, but it used to be that if you did something like NonQueryResult StoreUser(User user) { var sql = "INSERT... It would use string interpolation to fill out the properties
- dubbel 3y agoThat is the CWE that they identify, but the code seems to store the apparently unhashed password in the database on top of that?
- worksonmine 3y agoNot best practice? That's a very generous way to describe storing plaintext passwords in logs. I've seen this in the wild too but that's no excuse.
- chinathrow 3y ago> I've seen this in the wild too but that's no excuse. See, the LLM also saw it in the wild...
- azangru 3y agoThis is where one needs a hyphen :-)
- siva7 3y agoA related headline could be "Security weaknesses of code produced by a junior developer". It says copilot in the product name -> it's not intended to replace the pilots (aka developers) brain.