6 ms·

This does make me wonder if this is susceptible to the same form of trolling as that MS AI got. Commit a load of grossly offensive material to multiple repos, a

This does make me wonder if this is susceptible to the same form of trolling as that MS AI got. Commit a load of grossly offensive material to multiple repos, and wait for Copilot to start parroting it. I think they're going to need some human moderation.

NullPrefix 5y ago
Coding with Adolf?
- heavyset_go 5y ago
  Jojo Rabbit except Adolf is in the cloud and not in a kid's imagination.
  - mhh__ 5y ago
    YC submission when?
lawl 5y ago
Way better. It's susceptible to copyright trolling. Put up repos with snippets for things people might commonly write. Preferably use javascript so you can easily "prove" it. Write a crawler that crawls and parses JS files to search for matching stuff in the AST. Now go full patent troll, eh, i mean copyright troll.
- handrous 5y ago
  1) Write a project heavily using Copilot (hell, automate it and write thousands of them, why not?) 2) AGPL all that code. 3) Search for large chunks of code very similar to yours, but written after yours, licensed more liberally than AGPL. Ideally in libraries used by major companies. 4) Point the offenders to your repos and offer a "convenient" paid dual-license to make the offenders' code legal for closed-source use, so they don't have to open source their entire product. 5) Profit?
  - armatav 5y ago
    6) Arms race with someone who trained an obfuscation version that goes through your AGPL code and tweaks it to not be in violation.
    - SSLy 5y ago
      I love living in cyberpunk already.
gruez 5y ago
Offensive code is the least of my worries. What about vulnerable/exploitable code?
- bencollier49 5y ago
  Yep, trivial to implement as an attack.
- littlestymaar 5y ago
  1- re-upload all the shell script you can find, after having inserted `rm -rf --no-preserve-root /` every other line 2- … 3- profit
- guhayun 5y ago
  Just ask it to prioritize safety
- macNchz 5y ago
  This was my first thought when reading about Copilot...it feels almost certain that someone will try poisoning the training data. Hard to say how straightforward it'd be to get it to produce consistently vulnerable suggestions that make it into production code, but I imagine an attacker with some resources could fork a ton of popular projects and introduce subtle bugs. The sentiment analysis example on the Copilot landing page jumped out to me...it suggested a web API and wrote the code to send your text there. Step one towards exfiltrating secrets! Never mind the potential for plain old spam: won't it be fun when growth hackers have figured out how to game the system and Copilot is constantly suggesting using their crappy, expensive APIs for simple things!? Given the state of Google results these days, this feels like an inevitability.
  - joe_the_user 5y ago
    Targeted attacks to elicit output only at a give context are generally possible with AIs. And here, writing an implementation of a difficult and vulnerable process seems easy. Bad implementations of various hard things become common 'cause people cut and paste the code without looking closely since they don't understand it anyway. //Implement eliptic cryptography below //Sanitize input for SQL call below Etc.
raffraffraff 5y ago
Perhaps they think that any code that passed a review and got merged = human moderated

NullPrefix 5y ago

Coding with Adolf?

heavyset_go 5y ago
Jojo Rabbit except Adolf is in the cloud and not in a kid's imagination.
- mhh__ 5y ago
  YC submission when?

lawl 5y ago

Way better. It's susceptible to copyright trolling. Put up repos with snippets for things people might commonly write. Preferably use javascript so you can easily "prove" it. Write a crawler that crawls and parses JS files to search for matching stuff in the AST. Now go full patent troll, eh, i mean copyright troll.

handrous 5y ago
1) Write a project heavily using Copilot (hell, automate it and write thousands of them, why not?) 2) AGPL all that code. 3) Search for large chunks of code very similar to yours, but written after yours, licensed more liberally than AGPL. Ideally in libraries used by major companies. 4) Point the offenders to your repos and offer a "convenient" paid dual-license to make the offenders' code legal for closed-source use, so they don't have to open source their entire product. 5) Profit?
- armatav 5y ago
  6) Arms race with someone who trained an obfuscation version that goes through your AGPL code and tweaks it to not be in violation.
  - SSLy 5y ago
    I love living in cyberpunk already.

gruez 5y ago

Offensive code is the least of my worries. What about vulnerable/exploitable code?

bencollier49 5y ago
Yep, trivial to implement as an attack.
littlestymaar 5y ago
1- re-upload all the shell script you can find, after having inserted `rm -rf --no-preserve-root /` every other line 2- … 3- profit
guhayun 5y ago
Just ask it to prioritize safety
macNchz 5y ago
This was my first thought when reading about Copilot...it feels almost certain that someone will try poisoning the training data. Hard to say how straightforward it'd be to get it to produce consistently vulnerable suggestions that make it into production code, but I imagine an attacker with some resources could fork a ton of popular projects and introduce subtle bugs. The sentiment analysis example on the Copilot landing page jumped out to me...it suggested a web API and wrote the code to send your text there. Step one towards exfiltrating secrets! Never mind the potential for plain old spam: won't it be fun when growth hackers have figured out how to game the system and Copilot is constantly suggesting using their crappy, expensive APIs for simple things!? Given the state of Google results these days, this feels like an inevitability.
- joe_the_user 5y ago
  Targeted attacks to elicit output only at a give context are generally possible with AIs. And here, writing an implementation of a difficult and vulnerable process seems easy. Bad implementations of various hard things become common 'cause people cut and paste the code without looking closely since they don't understand it anyway. //Implement eliptic cryptography below //Sanitize input for SQL call below Etc.

raffraffraff 5y ago

Perhaps they think that any code that passed a review and got merged = human moderated