4 ms·

In light of this, unless there is evidence to the contrary, I'm going to assume it will also regurgitate malicious code.

by lokl 5y ago

qayxc 5y ago
> I'm going to assume it will also regurgitate malicious code. Well if it wouldn't replicate or even randomly generate malicious code, it would imply that CoPilot would somehow be able to solve Halting Problem or - at the very least - understand intent and purpose of both its output and training material. Keep in mind that the very definition of "malicious code" is highly subjective, plus the intent and purpose aren't necessarily encoded in the program itself. If the latter were the case, there would be no need for documentation, requirements or specs.
- lokl 5y ago
  When I say "malicious code," what I really mean is some well-known patterns of malicious code, not all malicious code in general. Just like we are surprised about "secrets" being regurgitated when we mean "API keys."
lennoff 5y ago
Oh, it does... It generates code that's vulnerable to SQL injections, XSS, etc. It was trained on such code! (hopefully this will improve)
- lokl 5y ago
  Training on code that unintentionally has vulnerabilities is a problem, but I'm even more worried about bad actors intentionally putting code with vulnerabilities on GitHub with the hope that it will become training data. Bad actors might learn how to disguise code to sneak it into Copilot (if disguise is even necessary) and introduce backdoors, etc. It could be especially dangerous because of the "stamp of approval" Copilot has from GitHub/Microsoft. People who would not copy/paste code from the web might feel a false sense of security using Copilot.
  - TeMPOraL 5y ago
    I totally expect this will happen. I bet this is already happening. As long as they continue to train Copilot on third-party code that wasn't thoroughly, manually vetted by them, this vector of attack will remain open - and mitigating it falls into the domain of... spam filtering.

In light of this, unless there is evidence to the contrary, I'm going to assume it will also regurgitate malicious code.

by lokl 5y ago

In light of this, unless there is evidence to the contrary, I'm going to assume it will also regurgitate malicious code.

qayxc 5y ago

> I'm going to assume it will also regurgitate malicious code. Well if it wouldn't replicate or even randomly generate malicious code, it would imply that CoPilot would somehow be able to solve Halting Problem or - at the very least - understand intent and purpose of both its output and training material. Keep in mind that the very definition of "malicious code" is highly subjective, plus the intent and purpose aren't necessarily encoded in the program itself. If the latter were the case, there would be no need for documentation, requirements or specs.

lokl 5y ago
When I say "malicious code," what I really mean is some well-known patterns of malicious code, not all malicious code in general. Just like we are surprised about "secrets" being regurgitated when we mean "API keys."

lennoff 5y ago

Oh, it does... It generates code that's vulnerable to SQL injections, XSS, etc. It was trained on such code! (hopefully this will improve)

lokl 5y ago
Training on code that unintentionally has vulnerabilities is a problem, but I'm even more worried about bad actors intentionally putting code with vulnerabilities on GitHub with the hope that it will become training data. Bad actors might learn how to disguise code to sneak it into Copilot (if disguise is even necessary) and introduce backdoors, etc. It could be especially dangerous because of the "stamp of approval" Copilot has from GitHub/Microsoft. People who would not copy/paste code from the web might feel a false sense of security using Copilot.
- TeMPOraL 5y ago
  I totally expect this will happen. I bet this is already happening. As long as they continue to train Copilot on third-party code that wasn't thoroughly, manually vetted by them, this vector of attack will remain open - and mitigating it falls into the domain of... spam filtering.