11 ms·

GitHub Copilot regurgitates valid secrets

201 points by petulla 5y ago

aloisdg 5y ago
This is why environment variables exist
- corobo 5y ago
  Definitely true but in this case it does add another point against "it generates code, not copies it"
  - the8472 5y ago
    Deliberate, repeated prompt engineering to make it regurgitate something is not proof for the general case. The same way that humans being able to recite text when asked to is not proof that they're incapable of creating new content.
    - corobo 5y ago
      To clarify my issue was not that it outputted quake prompted -- course that's obvious, it's clear the AI had no intention of outputting the code until the user provided function Q_ The issue is that it's impossible to tell if it's done that or not short of googling each line it comes out with Anyway I've determined my reactions are now based on disappointment rather than thought, a bias that should be factored in
- rightbyte 5y ago
  Which you put in a startup.sh, which you later commit. With a lesson learned you add those to git ignore. Speaking for a friend.
the8472 5y ago
On the other hand it also means someone checked those secrets into github somewhere, so they would also be retrievable with a classic search.
- henearkr 5y ago
  Has Copilot been trained on private repos as well? If so, it means that you wouldn't find them by a search, but they would still be revealed by the A.I.
  - the8472 5y ago
    The question is easily answered by checking their FAQ: > What data has GitHub Copilot been trained on? > GitHub Copilot is powered by OpenAI Codex, a new AI system created by OpenAI. It has been trained on a selection of English language and source code from publicly available sources, including code in public repositories on GitHub.
    - henearkr 5y ago
      Thanks and sorry to be so lazy (facepalm)...
    - intricatedetail 5y ago
      Who can you verify it is true?
      - mrfusion 5y ago
        We trust corporations to tell us the truth every day. What do you think the news is.
        na85 5y ago
        Everybody knows the news can be charitably described as inaccurate. Sure, Microsoft Marketing Department says it was trained on public code. The PM probably told them it was. Was it actually trained on public code only?
        paavohtl 5y ago
        I really don't think that's realistic. They would get sued by every single one of their major customers for leaking confidential information.
sergiomattei 5y ago
Called it.
- BtM909 5y ago
  Can you get these keys by searching via a search engine?
  - erikrothoff 5y ago
    Yes. There are quite a few services that scan Github and in realtime show leaked secrets. I once found someones Gmail password that way...
  - karmasimida 5y ago
    CommonCrawl contains considerable amount of PII information if you really spend time digging, for example.
- nojito 5y ago
  The keys would be sniffed out by secret scanner anyway https://docs.github.com/en/code-security/secret-security/about-secret-scanning https://docs.github.com/en/code-security/secret-security/abo... Doesn’t make sense to implement fixes on CoPilot.
  - atraac 5y ago
    The stuff you write doesn't have to be commited to GitHub(or am I missing something?) so this argument makes no sense. Copilot clearly scanned and autocompleted third party secrets, it's in no way acceptable behaviour.
    - nojito 5y ago
      It's trained on public repos. Public repos are scanned by the secret scanner. Why is it CoPilot's responsibiity to also scan the keys too?
    - dolmen 5y ago
input_sh 5y ago
There truly is an XKCD for everything: https://xkcd.com/2169/ https://xkcd.com/2169/
rvz 5y ago
So GitHub Copilot has inherited all the bad practices of many StackOverFlow and GitHub side projects and generates them in front of you as 'assistance'. All the API keys are still working and who knows, someone might complain about a huge fee right in here because they forgot to revoke it. Only time will tell. I am certainly going to avoid this contraption. No thanks and most certainly no deal. Downvoters: So are you saying GitHub Copilot DOES NOT do the following: Leak working API keys in the editor. Generate broken code AND give you the wrong implementation if you add a single typo? Copy and regurgitates copyrighted code verbatim. Guesses 1 out of 10 tries. Send parts of your code when you type in the editor. Are you VERY sure?
- dolmen 5y ago
  > Leak working API keys in the editor. The content of your editor is sent by Copilot to its cloud service (the FAQ says: "The GitHub Copilot editor extension sends your comments and code to the GitHub Copilot service"). So yes any editor content is leaked, including sensitive information. But is this content sent to other Copilot users? AFAIK, no. The FAQ mentions the OpenAI Codex as the training set. https://openai.com/ https://openai.com/
  - dolmen 5y ago
    I just had a look at the OpenAI website. They advertise an English-to-French translation service powered by AI. But it appears that nobody who is French native has even reviewed the service presentation. When the marketing material is just a joke, what can you expect in production if as a customer you use the service? Just this example tells me much about the internal organization of the company.
phumberdroz 5y ago
The only time I would think this is a valid security issue if those were tokens that were previously not public. But that should not be the case right?
- holstvoogd 5y ago
  Lets hope so... I expect that these were accidentally committed to a public repo However, while the keys are then already leaked, you'd have to go search for them. Copilot suggests you use them in you editor. That is not quite the same imo. It goes from deliberately searching and using leaked keys to having them handed to you without context. I feel it is a bit like finding an unlocked bike, if you take it, it is still stealing. But here there is a guy at the bike parking lets say that is handing out bikes to anyone passing by. Not the best analogy, but i think it covers my point ;)
  - phumberdroz 5y ago
    I think it would be more like a friend telling you to take the bike or saying it is his bike and you can take it for a ride. But yes I get your point but I also believe people still need to apply some sense to what co pilot suggests.
- e12e 5y ago
  Sure, if someone checked in a secret to a repo that at some point was public, and got crawled by co-pilot - they should cycle that secret, so it's no longer valid - rather than only mark the repo private and/or nuke the secret from the repo history. But there's another side to this - if you write code using co-pilot against a popular Api - and co-pilot gives you a valid key - and you access data or a system you aren't supposed to - would you be liable under the various draconian antighacker laws? If you pick up a key card from the street, and enter someone's home - you'd be trespassing after all..
  - phumberdroz 5y ago
    That is a good question and I think you should be. After all you are still the Person that writes and produces the code just with the help of a tool. Similar to a lockpick. (I hope that makes sense)
notimrelakatos 5y ago
I look forward to the bright future were I have to maintain messy code from AI Rockstars.
- weird-eye-issue 5y ago
  This is exactly what I'm afraid of. If people use this tool early in their journey of learning programming it will do them a big disservice.
- avipars 5y ago
  The next generation of script kiddies
aritmo 5y ago
It is one thing to put by accident your API key on your public Github repository. And it's another (bigger) issue for Copilot to pick up that API key and put it in someone else's project.
- blagie 5y ago
  No. Your public leak is the bigger issue. Copilot is an issue, but I would assume malicious agents are already doing copilot-like things, just not in the open.
iamlucaswolf 5y ago
What amazes me is how predictable(?) all of the recent issues were. Don't get me wrong, the folks behind Copilot are clearly, without any doubt smart, creative, and capable. But then... None of these issues (reproducing licensed code ad verbatim, non-compiling code, getting semantics wrong, and now this) are 0.01% edge cases that take specialized knowledge to see or trigger. I remember some of them being called days ago in the initial HN thread by people who haven't had beta access. I really wonder how this announcement/rollout looked like on the management side of things. Because a) these shortcomings must have been known beforehand and b) backlash from people who feel threatened for their jobs/"stolen" of their open source work was (I guess) foreseeable? I've already read calls to abandon GitHub for competitors; this can hardly have been an acceptable outcome here. Nevertheless, Copilot is still one of the most innovative and interesting products I've seen in a while.
- callamdelaney 5y ago
  There are things that already do what Copilot does, eg Kite so it's hardly innovative imo.
  - nicce 5y ago
    Kite does only fraction what Copilot is currently doing. It is great at suggesting function names and parameters, but it does not really suggest complete code or generate somewhat new code.
- syshum 5y ago
  I dont know if that true. Just because you are "smart, creative, and capable" does not mean you can predict every possible outcome or be incapable of missing the obvious I have been on both sides of that, where I have had to point out obvious flaws in an idea to very smart people, and have had clearly obvious flaw pointed out to me in one of my idea's... I think it is completely possible that some or even all of the issues co-pilot is facing were unknown at the time of release, even if they are obvious to some
  - tyingq 5y ago
s_gourichon 5y ago
It does not generate secrets. The Twitter conversation does not mention that word. Most certainly, it regurgitates secrets it has seen on crawled repos. Can the title be adjusted, please?
- deleted 5y ago
  [deleted]
- chrisseaton 5y ago
  I think ‘generates’ means produces this output in this context. It’s the correct term for the technology being talked about. Nobody is confused that it’s producing new cryptographic tokens.
  - playpause 5y ago
    I was confused. "Copilot generates valid secrets" sounds like it can be used to generate new secrets that are valid in format or something. The headline is misleading, even if you personally weren't misled. The secrets are not being generated, they are real secrets appearing in generated code.
    - chrisseaton 5y ago
      > The secrets are not being generated They are being generated. It's a Generative Pre-trained Transformer. It's generative. It generates things. That's what this technical term means. It's correct.
      - captaincaveman 5y ago
        Terms have different meaning in different context, I think both terms are valid in this case, thus a qualifying word to remove ambiguity is the normal solution.
      - alkonaut 5y ago
        I think a generative transformer can be said to generate text. It doesn't generate new words. The words it uses to generate the text, are words copied verbatim from the input. What's emergent is the combination of those words into text. The generated output has building blocks (characters, words) which themselves are not "generated" in the sense that they are novel. A random number generator generates numbers. It doesn't generate the digits used to represent the random numbers. Those are taken from a set (such as 0..9) and just "used".
0x0 5y ago
Any chance Copilot could be made to cough up the DVDCSS or BluRay AACS DRM secrets?
rsynnott 5y ago
I'm kind of astonished that this project got greenlit, given Microsoft's previous experiences with embarrassing AI projects (thinking particularly of Tay and Zo).
- blagie 5y ago
  Microsoft isn't a person. Individuals at organizations make decisions, and it's very possible the person greenlighting this never heard of Tay and Zo. It's almost certain they weren't the same person.
  - account42 5y ago
    You think it's reasonable for someone greenlighting projects at MS to not familiarize themselves with releated previous projects?
    - rsynnott 5y ago
      And, particularly, extremely high profile, embarrassing projects in the same general space?
    - blagie 5y ago
      My comment wasn't about reasonableness. My comment was about organizational reality. How many times have you seen the two coincide in large orgs, especially ones as complex as Microsoft? People treat organizational decisions from large orgs as if they were made by a particularly stupid, incompetent individual, but that's not what happens. They're made by organizational processes, incentive structures, and emergent behaviors. That's not cynicism -- structuring over 100,000 people to collectively act in ways one might consider reasonable is a genuinely hard problem. We tend to blame unreasonable people or malicious behavior, but every person in an organization can be smart, ethical, and reasonable, and stupid stuff will still happen. If there were one bad apple, like Enron, it'd be fine to blame the people there. If it's nearly every large organization in the history of humanity -- including formerly good ones like Google -- it's reasonable to look for a more systemic explanation than unreasonable people.
deleted 5y ago
[deleted]
lokl 5y ago
In light of this, unless there is evidence to the contrary, I'm going to assume it will also regurgitate malicious code.
- qayxc 5y ago
  > I'm going to assume it will also regurgitate malicious code. Well if it wouldn't replicate or even randomly generate malicious code, it would imply that CoPilot would somehow be able to solve Halting Problem or - at the very least - understand intent and purpose of both its output and training material. Keep in mind that the very definition of "malicious code" is highly subjective, plus the intent and purpose aren't necessarily encoded in the program itself. If the latter were the case, there would be no need for documentation, requirements or specs.
  - lokl 5y ago
    When I say "malicious code," what I really mean is some well-known patterns of malicious code, not all malicious code in general. Just like we are surprised about "secrets" being regurgitated when we mean "API keys."
- lennoff 5y ago
  Oh, it does... It generates code that's vulnerable to SQL injections, XSS, etc. It was trained on such code! (hopefully this will improve)
  - lokl 5y ago
    Training on code that unintentionally has vulnerabilities is a problem, but I'm even more worried about bad actors intentionally putting code with vulnerabilities on GitHub with the hope that it will become training data. Bad actors might learn how to disguise code to sneak it into Copilot (if disguise is even necessary) and introduce backdoors, etc. It could be especially dangerous because of the "stamp of approval" Copilot has from GitHub/Microsoft. People who would not copy/paste code from the web might feel a false sense of security using Copilot.
llimos 5y ago
I do feel for the people behind Copilot, even though they'll have known it was coming. They produce something absolutely frggin' amazing that can change the world and for the next few days all everyone does is pile on and pull it to pieces... yes of course these are valid issues but can we please look at the big picture and appreciate what an achievement this is?
- esailija 5y ago
  It's only because people like you pretend this glorified markov chain is something more that is causing everyone to pile on. So stop.
  - treesprite82 5y ago
    > you pretend this glorified markov chain is something more I hate this belittling attitude that HN has towards projects in certain fields. It's like if I dismissed any progress in graphics as "just glorified triangle-drawing".
- cdrini 5y ago
  I agree with this; I'm very confused by what appears to be a very strong visceral reaction to this experimental feature. I don't know what impact it will have on programmers/programming, but I'm curious to see. Personally, I see something like copilot as a terrific search engine. Searching for code in Github is kind of difficult; being able to search by writing a descriptive comment is really cool!
- TomSwirly 5y ago
  > They produce something absolutely frggin' amazing that can change the world It won't change nothing. It's madlibs for code. The 90% of programmers who are mediocre will drown out the 9% who are competent and the 1% who are talented.
e12e 5y ago
> SendGrid engineer reports API keys generated by the AI are not only valid but still functional. > GitHub CEO acknowledges the issue... still waiting for them to pull the plug I agree this is an issue for co-pilot as well - but it's really on send grid to invalidate keys that are known to be leaked? Yes, that's inconvenient for the affected customers - otoh they won't get billed for other people's usage - or dinged for someone spamming using their keys...
- karmicthreat 5y ago
  Most sendgrid customers are in their public IP pool. They don't tolerate spam on those IPs because its difficult to manage with the spam lists. So they are definitely proactive about killing leaked keys. I leaked one when I accidentally left one in a repo I was making public. Took 15 minutes for sendgrid to drop it after putting the repo public.
villgax 5y ago
We are in a time where you have a crystal ball/chip & who can whisper sweet nothings & get back answers
WillDaSilva 5y ago
I don't consider this a problem. Copilot was trained on public repos, so these secrets had to be checked into public repos. They were already totally public, and should have been invalidated/replaced and redacted. Copilot might result in previously undiscovered published secrets being found, but that's not much worse than anyone finding one under normal circumstances.
- intricatedetail 5y ago
  Have they produced evidence it was trained only on public repos? They should release the model and tooling so we can verify that.
  - treesprite82 5y ago
    The easiest way to test this would probably be to try to get it to generate code/secrets that appear only in private repos.
- deleted 5y ago
  [deleted]
MontyCarloHall 5y ago
Unintentional copyright violations and “leaking” of secrets people accidentally committed to public repos aside, my main issue with Copilot is that I don’t think it actually makes coding easier. Everyone knows it’s usually far easier to write code than to read code. Writing code is a nonlinear process: you don’t start from the first character and write everything out in one single pass. Instead, the logic of the code evolves nonlinearly—add a bit here, remove a bit there, restructure a bit over there. Good code is written such that it can be mostly understood in a single pass, but this is not always possible. For example, understanding function calls requires jumping around the code to where the function is defined (and often deeper down the stack). Understanding a conditional with multiple branches requires first reading all the conditional predicates before reading the code blocks they lead to. Reading, on the other hand, is naturally a linear process. Understanding code requires reconstructing the nonlinear flow though it, and the nonlinear thought process used to write it in the first place. This is why constant communication between partners during pair programming is essential—if too much unexplained code gets dumped on a partner, figuring out how it works takes longer than just writing it themself. Copilot is like pair programming with a completely incommunicative partner who can’t walk you through the code they just wrote. You therefore still have to review most of it manually, which takes much longer than writing it yourself in the first place.
- whydid 5y ago
  In theory, I agree with your point. However, I've worked with people who struggle to write in English without introducing random punctuation, and don't "see" (or care about) the text on the screen enough to go back and fix it. I think Copilot will be a great benefit to the lazy programmer, who understands the semantics, but just can't be bothered to get the indentation or other syntax correct.
  - MontyCarloHall 5y ago
    I 100% agree, but that’s exactly what code linters do, which have been around for decades. That said, a more sophisticated linter might be useful in catching non-idiomatic, but syntactically/stylistically valid code that would thus be flagged as “valid” by current linters’ simple automata.
intricatedetail 5y ago
Grand Source Code theft. A permanent stain on Github? They should scrap it and Microsoft should be ordered to sell Github because they have a conflict of interest. For example Microsoft has access to your private repos and can do things like co pilot with your data. Who knows maybe your code powers Windows 11 now.
- easton 5y ago
  Just like the conflict of interest they’ve had for 20 years selling TFS/Azure DevOps?
beshrkayali 5y ago
It's really kind of comical at this point. The more this copilot bs continues to be a thing, the more it's making Github seem irresponsible/careless at best.
- e3bc54b2 5y ago
  The cynic in me is thinking that marketing folks at Microsoft/Github are giggling endlessly at all the stories giving them free and extreme publicity. This is enforced by the recent post by Github 'analyzing' the copilot's code regurgitation, instead of retracting and retraining it on more defined subset of codebases. This thing is worthless at best, annoying at everything and terrifyingly capable of destroying every programmer's productivity at worst. But it will stay, it will grow because stupid execs will keep dreaming of replacing their engineering talent with this, and Microsoft will laugh all the way to the bank.
  - WesolyKubeczek 5y ago
    Just until the software at the bank is rewritten using CoPilot and suddenly they can’t withdraw any money.
  - beshrkayali 5y ago
    There's no way they didn't expect some backlash on this. So I think it's partly for the marketing gimmick. I'm sure there are people at Github tho who really think they're making something valuable unfortunately. Sadly, what they're not aware of is that they've become part of the experiment themselves. It's like expecting that an AI trained on Shakespeare novels would be able to "help" a writer write Shakespeare-like novels. Sure, they might get something that might fool some people, but are they a writer? I think software is a lot more like "writing" than it is like "building". What mostly annoys me is that this is a win-win for github regardless of the outcome. If people buy into it (even for just a while, and it currently seems like some really smart people are buying into it) they'll carve a huge piece of a new market. If it fails, they'll make it seem like an experiment into the whole ethical gray area of what should and shouldn't be used for training, and that they just wanted to draw attention to it.
tyingq 5y ago
I wish he would have tried to track down if the keys were in a public repo before asking Sendgrid about them. If they turned out to be only on Github private repos, that would be new and interesting info. Not saying putting keys in a private, but 3rd party hosted repo, is a terrific idea.
cabirum 5y ago
Can we please stop (mis)using the term "AI"? It just does not live up to most people's expectations. Copilot is a glorified Markov chain autocomplete sitting on a huge dump pile of data. It is not aware of constructs such as "licenses" or "secrets" most people would have expected from AI. To prevent it from spilling secrets everywhere, a developer ~~should teach the AI a concept of secrets and the meaning of licenses.~~ has to implement a filter. A regexp-based one will do, I guess.
- armatav 5y ago
  Yeah, this. Modern AI is not AI - AI is synthetic machine life and essentially always has been, in fiction and non-fictional idealism. Deep neural networks have stuck us in hope-fueled uncanny valley, and very smart people tend to become very confused about their technology when they're subject to it. This technology has its place about heuristic programming, definitely, but is not AI.
  - TeMPOraL 5y ago
    Someone will definitely be quick to reply with the "AI is a moving goalpost, things stop being called AI when they work, for example..." - so I'll offer my counterpoint up front. These things were never AI except in marketing lingo and in connection to the research in machine learning. The common folk definition of AI doesn't change - it's still the same vision of computers in science fiction, with which you can converse, and which can think better than you (except in some specific ways in which are super-dumb - this is necessary for the story to have any plot).
    - armatav 5y ago
      Right. And just to be clear I love the field and most everybody in it, specifically for their idealism - I'm a practitioner myself - clearly everyone involved in the modern field of AI wants to leave a legacy of beneficial impact, and sees AI as their tool to do so. I just think if we're looking for life we won't find it in the gates of a transistor.
    - randallsquared 5y ago
fxtentacle 5y ago
If Copilot was trained only on public repos like they claim, then shouldn't those API keys already be disabled due to existing secret scanning tools? For example https://docs.github.com/en/code-security/secret-security/about-secret-scanning https://docs.github.com/en/code-security/secret-security/abo... The fact that Copilot recreates API keys that still work makes me wonder if they come from a semi-public place, because SendGrid is usually quite fast at blocking API keys that were accidentally made public.
sputknick 5y ago
I see this as a problem with the developers who are committing code, and not a problem with Copilot. if you make your secrets accessible then they might be accessed. Also if you are rotating your keys regularly that would also mitigate these issues. This is a problem with humans failing to execute known security best practices, not malicious AI doing something insidious.
speedgoose 5y ago
People put valid secrets in their public repository all the time. Just a quick search: https://grep.app/search?q=%28secret%7Capi%29_%3Fkey%5Cs%3A%3F%3D%20%5B%22%27%5D%5Ba-zA-Z0-9%5D%7B8%2C%7D%22&regexp=true https://grep.app/search?q=%28secret%7Capi%29_%3Fkey%5Cs%3A%3...
tgsovlerkhgsel 5y ago
Tweet taken down (?), does anyone have a mirror?
- deleted 5y ago
  [deleted]
mvolfik 5y ago
https://web.archive.org/web/20210705123028/https://twitter.com/alexjc/status/1411966249437995010 https://web.archive.org/web/20210705123028/https://twitter.c... > COPILOT SECURITY BREACH > SendGrid engineer reports API keys generated by the AI are not only valid but still functional. > GitHub CEO acknowledges the issue... still waiting for them to pull the plug or make a comment. :popcorn: Quoting https://twitter.com/pkell7/status/1411058236321681414 https://twitter.com/pkell7/status/1411058236321681414
ibraheemdev 5y ago
Was the tweet taken down?
Paradigma11 5y ago
I really dont think that stuff hosted in public repos can be classified as secrets.

201 points by petulla 5y ago

aloisdg 5y ago

This is why environment variables exist

corobo 5y ago
Definitely true but in this case it does add another point against "it generates code, not copies it"
- the8472 5y ago
  Deliberate, repeated prompt engineering to make it regurgitate something is not proof for the general case. The same way that humans being able to recite text when asked to is not proof that they're incapable of creating new content.
  - corobo 5y ago
    To clarify my issue was not that it outputted quake prompted -- course that's obvious, it's clear the AI had no intention of outputting the code until the user provided function Q_ The issue is that it's impossible to tell if it's done that or not short of googling each line it comes out with Anyway I've determined my reactions are now based on disappointment rather than thought, a bias that should be factored in
rightbyte 5y ago
Which you put in a startup.sh, which you later commit. With a lesson learned you add those to git ignore. Speaking for a friend.

the8472 5y ago

On the other hand it also means someone checked those secrets into github somewhere, so they would also be retrievable with a classic search.

henearkr 5y ago
Has Copilot been trained on private repos as well? If so, it means that you wouldn't find them by a search, but they would still be revealed by the A.I.
- the8472 5y ago
  The question is easily answered by checking their FAQ: > What data has GitHub Copilot been trained on? > GitHub Copilot is powered by OpenAI Codex, a new AI system created by OpenAI. It has been trained on a selection of English language and source code from publicly available sources, including code in public repositories on GitHub.
  - henearkr 5y ago
    Thanks and sorry to be so lazy (facepalm)...
  - intricatedetail 5y ago
    Who can you verify it is true?
    - mrfusion 5y ago
      We trust corporations to tell us the truth every day. What do you think the news is.
      - na85 5y ago
        Everybody knows the news can be charitably described as inaccurate. Sure, Microsoft Marketing Department says it was trained on public code. The PM probably told them it was. Was it actually trained on public code only?
        paavohtl 5y ago
        I really don't think that's realistic. They would get sued by every single one of their major customers for leaking confidential information.

sergiomattei 5y ago

Called it.

BtM909 5y ago
Can you get these keys by searching via a search engine?
- erikrothoff 5y ago
  Yes. There are quite a few services that scan Github and in realtime show leaked secrets. I once found someones Gmail password that way...
- karmasimida 5y ago
  CommonCrawl contains considerable amount of PII information if you really spend time digging, for example.
nojito 5y ago
The keys would be sniffed out by secret scanner anyway https://docs.github.com/en/code-security/secret-security/about-secret-scanning https://docs.github.com/en/code-security/secret-security/abo... Doesn’t make sense to implement fixes on CoPilot.
- atraac 5y ago
  The stuff you write doesn't have to be commited to GitHub(or am I missing something?) so this argument makes no sense. Copilot clearly scanned and autocompleted third party secrets, it's in no way acceptable behaviour.
  - nojito 5y ago
    It's trained on public repos. Public repos are scanned by the secret scanner. Why is it CoPilot's responsibiity to also scan the keys too?
  - dolmen 5y ago

input_sh 5y ago

There truly is an XKCD for everything: https://xkcd.com/2169/ https://xkcd.com/2169/

rvz 5y ago

So GitHub Copilot has inherited all the bad practices of many StackOverFlow and GitHub side projects and generates them in front of you as 'assistance'. All the API keys are still working and who knows, someone might complain about a huge fee right in here because they forgot to revoke it. Only time will tell. I am certainly going to avoid this contraption. No thanks and most certainly no deal. Downvoters: So are you saying GitHub Copilot DOES NOT do the following: Leak working API keys in the editor. Generate broken code AND give you the wrong implementation if you add a single typo? Copy and regurgitates copyrighted code verbatim. Guesses 1 out of 10 tries. Send parts of your code when you type in the editor. Are you VERY sure?

dolmen 5y ago
> Leak working API keys in the editor. The content of your editor is sent by Copilot to its cloud service (the FAQ says: "The GitHub Copilot editor extension sends your comments and code to the GitHub Copilot service"). So yes any editor content is leaked, including sensitive information. But is this content sent to other Copilot users? AFAIK, no. The FAQ mentions the OpenAI Codex as the training set. https://openai.com/ https://openai.com/
- dolmen 5y ago
  I just had a look at the OpenAI website. They advertise an English-to-French translation service powered by AI. But it appears that nobody who is French native has even reviewed the service presentation. When the marketing material is just a joke, what can you expect in production if as a customer you use the service? Just this example tells me much about the internal organization of the company.

phumberdroz 5y ago

The only time I would think this is a valid security issue if those were tokens that were previously not public. But that should not be the case right?

holstvoogd 5y ago
Lets hope so... I expect that these were accidentally committed to a public repo However, while the keys are then already leaked, you'd have to go search for them. Copilot suggests you use them in you editor. That is not quite the same imo. It goes from deliberately searching and using leaked keys to having them handed to you without context. I feel it is a bit like finding an unlocked bike, if you take it, it is still stealing. But here there is a guy at the bike parking lets say that is handing out bikes to anyone passing by. Not the best analogy, but i think it covers my point ;)
- phumberdroz 5y ago
  I think it would be more like a friend telling you to take the bike or saying it is his bike and you can take it for a ride. But yes I get your point but I also believe people still need to apply some sense to what co pilot suggests.
e12e 5y ago
Sure, if someone checked in a secret to a repo that at some point was public, and got crawled by co-pilot - they should cycle that secret, so it's no longer valid - rather than only mark the repo private and/or nuke the secret from the repo history. But there's another side to this - if you write code using co-pilot against a popular Api - and co-pilot gives you a valid key - and you access data or a system you aren't supposed to - would you be liable under the various draconian antighacker laws? If you pick up a key card from the street, and enter someone's home - you'd be trespassing after all..
- phumberdroz 5y ago
  That is a good question and I think you should be. After all you are still the Person that writes and produces the code just with the help of a tool. Similar to a lockpick. (I hope that makes sense)

notimrelakatos 5y ago

I look forward to the bright future were I have to maintain messy code from AI Rockstars.

weird-eye-issue 5y ago
This is exactly what I'm afraid of. If people use this tool early in their journey of learning programming it will do them a big disservice.
avipars 5y ago
The next generation of script kiddies

aritmo 5y ago

It is one thing to put by accident your API key on your public Github repository. And it's another (bigger) issue for Copilot to pick up that API key and put it in someone else's project.

blagie 5y ago
No. Your public leak is the bigger issue. Copilot is an issue, but I would assume malicious agents are already doing copilot-like things, just not in the open.

iamlucaswolf 5y ago

What amazes me is how predictable(?) all of the recent issues were. Don't get me wrong, the folks behind Copilot are clearly, without any doubt smart, creative, and capable. But then... None of these issues (reproducing licensed code ad verbatim, non-compiling code, getting semantics wrong, and now this) are 0.01% edge cases that take specialized knowledge to see or trigger. I remember some of them being called days ago in the initial HN thread by people who haven't had beta access. I really wonder how this announcement/rollout looked like on the management side of things. Because a) these shortcomings must have been known beforehand and b) backlash from people who feel threatened for their jobs/"stolen" of their open source work was (I guess) foreseeable? I've already read calls to abandon GitHub for competitors; this can hardly have been an acceptable outcome here. Nevertheless, Copilot is still one of the most innovative and interesting products I've seen in a while.

callamdelaney 5y ago
There are things that already do what Copilot does, eg Kite so it's hardly innovative imo.
- nicce 5y ago
  Kite does only fraction what Copilot is currently doing. It is great at suggesting function names and parameters, but it does not really suggest complete code or generate somewhat new code.
syshum 5y ago
I dont know if that true. Just because you are "smart, creative, and capable" does not mean you can predict every possible outcome or be incapable of missing the obvious I have been on both sides of that, where I have had to point out obvious flaws in an idea to very smart people, and have had clearly obvious flaw pointed out to me in one of my idea's... I think it is completely possible that some or even all of the issues co-pilot is facing were unknown at the time of release, even if they are obvious to some
- tyingq 5y ago

s_gourichon 5y ago

It does not generate secrets. The Twitter conversation does not mention that word. Most certainly, it regurgitates secrets it has seen on crawled repos. Can the title be adjusted, please?

deleted 5y ago
[deleted]
chrisseaton 5y ago
I think ‘generates’ means produces this output in this context. It’s the correct term for the technology being talked about. Nobody is confused that it’s producing new cryptographic tokens.
- playpause 5y ago
  I was confused. "Copilot generates valid secrets" sounds like it can be used to generate new secrets that are valid in format or something. The headline is misleading, even if you personally weren't misled. The secrets are not being generated, they are real secrets appearing in generated code.
  - chrisseaton 5y ago
    > The secrets are not being generated They are being generated. It's a Generative Pre-trained Transformer. It's generative. It generates things. That's what this technical term means. It's correct.
    - captaincaveman 5y ago
      Terms have different meaning in different context, I think both terms are valid in this case, thus a qualifying word to remove ambiguity is the normal solution.
    - alkonaut 5y ago
      I think a generative transformer can be said to generate text. It doesn't generate new words. The words it uses to generate the text, are words copied verbatim from the input. What's emergent is the combination of those words into text. The generated output has building blocks (characters, words) which themselves are not "generated" in the sense that they are novel. A random number generator generates numbers. It doesn't generate the digits used to represent the random numbers. Those are taken from a set (such as 0..9) and just "used".

0x0 5y ago

Any chance Copilot could be made to cough up the DVDCSS or BluRay AACS DRM secrets?

rsynnott 5y ago

I'm kind of astonished that this project got greenlit, given Microsoft's previous experiences with embarrassing AI projects (thinking particularly of Tay and Zo).

blagie 5y ago
Microsoft isn't a person. Individuals at organizations make decisions, and it's very possible the person greenlighting this never heard of Tay and Zo. It's almost certain they weren't the same person.
- account42 5y ago
  You think it's reasonable for someone greenlighting projects at MS to not familiarize themselves with releated previous projects?
  - rsynnott 5y ago
    And, particularly, extremely high profile, embarrassing projects in the same general space?
  - blagie 5y ago
    My comment wasn't about reasonableness. My comment was about organizational reality. How many times have you seen the two coincide in large orgs, especially ones as complex as Microsoft? People treat organizational decisions from large orgs as if they were made by a particularly stupid, incompetent individual, but that's not what happens. They're made by organizational processes, incentive structures, and emergent behaviors. That's not cynicism -- structuring over 100,000 people to collectively act in ways one might consider reasonable is a genuinely hard problem. We tend to blame unreasonable people or malicious behavior, but every person in an organization can be smart, ethical, and reasonable, and stupid stuff will still happen. If there were one bad apple, like Enron, it'd be fine to blame the people there. If it's nearly every large organization in the history of humanity -- including formerly good ones like Google -- it's reasonable to look for a more systemic explanation than unreasonable people.

deleted 5y ago

[deleted]

lokl 5y ago

In light of this, unless there is evidence to the contrary, I'm going to assume it will also regurgitate malicious code.

qayxc 5y ago
> I'm going to assume it will also regurgitate malicious code. Well if it wouldn't replicate or even randomly generate malicious code, it would imply that CoPilot would somehow be able to solve Halting Problem or - at the very least - understand intent and purpose of both its output and training material. Keep in mind that the very definition of "malicious code" is highly subjective, plus the intent and purpose aren't necessarily encoded in the program itself. If the latter were the case, there would be no need for documentation, requirements or specs.
- lokl 5y ago
  When I say "malicious code," what I really mean is some well-known patterns of malicious code, not all malicious code in general. Just like we are surprised about "secrets" being regurgitated when we mean "API keys."
lennoff 5y ago
Oh, it does... It generates code that's vulnerable to SQL injections, XSS, etc. It was trained on such code! (hopefully this will improve)
- lokl 5y ago
  Training on code that unintentionally has vulnerabilities is a problem, but I'm even more worried about bad actors intentionally putting code with vulnerabilities on GitHub with the hope that it will become training data. Bad actors might learn how to disguise code to sneak it into Copilot (if disguise is even necessary) and introduce backdoors, etc. It could be especially dangerous because of the "stamp of approval" Copilot has from GitHub/Microsoft. People who would not copy/paste code from the web might feel a false sense of security using Copilot.

llimos 5y ago

I do feel for the people behind Copilot, even though they'll have known it was coming. They produce something absolutely frggin' amazing that can change the world and for the next few days all everyone does is pile on and pull it to pieces... yes of course these are valid issues but can we please look at the big picture and appreciate what an achievement this is?

esailija 5y ago
It's only because people like you pretend this glorified markov chain is something more that is causing everyone to pile on. So stop.
- treesprite82 5y ago
  > you pretend this glorified markov chain is something more I hate this belittling attitude that HN has towards projects in certain fields. It's like if I dismissed any progress in graphics as "just glorified triangle-drawing".
cdrini 5y ago
I agree with this; I'm very confused by what appears to be a very strong visceral reaction to this experimental feature. I don't know what impact it will have on programmers/programming, but I'm curious to see. Personally, I see something like copilot as a terrific search engine. Searching for code in Github is kind of difficult; being able to search by writing a descriptive comment is really cool!
TomSwirly 5y ago
> They produce something absolutely frggin' amazing that can change the world It won't change nothing. It's madlibs for code. The 90% of programmers who are mediocre will drown out the 9% who are competent and the 1% who are talented.

e12e 5y ago

> SendGrid engineer reports API keys generated by the AI are not only valid but still functional. > GitHub CEO acknowledges the issue... still waiting for them to pull the plug I agree this is an issue for co-pilot as well - but it's really on send grid to invalidate keys that are known to be leaked? Yes, that's inconvenient for the affected customers - otoh they won't get billed for other people's usage - or dinged for someone spamming using their keys...

karmicthreat 5y ago
Most sendgrid customers are in their public IP pool. They don't tolerate spam on those IPs because its difficult to manage with the spam lists. So they are definitely proactive about killing leaked keys. I leaked one when I accidentally left one in a repo I was making public. Took 15 minutes for sendgrid to drop it after putting the repo public.

villgax 5y ago

We are in a time where you have a crystal ball/chip & who can whisper sweet nothings & get back answers

WillDaSilva 5y ago

I don't consider this a problem. Copilot was trained on public repos, so these secrets had to be checked into public repos. They were already totally public, and should have been invalidated/replaced and redacted. Copilot might result in previously undiscovered published secrets being found, but that's not much worse than anyone finding one under normal circumstances.

intricatedetail 5y ago
Have they produced evidence it was trained only on public repos? They should release the model and tooling so we can verify that.
- treesprite82 5y ago
  The easiest way to test this would probably be to try to get it to generate code/secrets that appear only in private repos.
deleted 5y ago
[deleted]

MontyCarloHall 5y ago

Unintentional copyright violations and “leaking” of secrets people accidentally committed to public repos aside, my main issue with Copilot is that I don’t think it actually makes coding easier. Everyone knows it’s usually far easier to write code than to read code. Writing code is a nonlinear process: you don’t start from the first character and write everything out in one single pass. Instead, the logic of the code evolves nonlinearly—add a bit here, remove a bit there, restructure a bit over there. Good code is written such that it can be mostly understood in a single pass, but this is not always possible. For example, understanding function calls requires jumping around the code to where the function is defined (and often deeper down the stack). Understanding a conditional with multiple branches requires first reading all the conditional predicates before reading the code blocks they lead to. Reading, on the other hand, is naturally a linear process. Understanding code requires reconstructing the nonlinear flow though it, and the nonlinear thought process used to write it in the first place. This is why constant communication between partners during pair programming is essential—if too much unexplained code gets dumped on a partner, figuring out how it works takes longer than just writing it themself. Copilot is like pair programming with a completely incommunicative partner who can’t walk you through the code they just wrote. You therefore still have to review most of it manually, which takes much longer than writing it yourself in the first place.

whydid 5y ago
In theory, I agree with your point. However, I've worked with people who struggle to write in English without introducing random punctuation, and don't "see" (or care about) the text on the screen enough to go back and fix it. I think Copilot will be a great benefit to the lazy programmer, who understands the semantics, but just can't be bothered to get the indentation or other syntax correct.
- MontyCarloHall 5y ago
  I 100% agree, but that’s exactly what code linters do, which have been around for decades. That said, a more sophisticated linter might be useful in catching non-idiomatic, but syntactically/stylistically valid code that would thus be flagged as “valid” by current linters’ simple automata.

intricatedetail 5y ago

Grand Source Code theft. A permanent stain on Github? They should scrap it and Microsoft should be ordered to sell Github because they have a conflict of interest. For example Microsoft has access to your private repos and can do things like co pilot with your data. Who knows maybe your code powers Windows 11 now.

easton 5y ago
Just like the conflict of interest they’ve had for 20 years selling TFS/Azure DevOps?

beshrkayali 5y ago

It's really kind of comical at this point. The more this copilot bs continues to be a thing, the more it's making Github seem irresponsible/careless at best.

e3bc54b2 5y ago
The cynic in me is thinking that marketing folks at Microsoft/Github are giggling endlessly at all the stories giving them free and extreme publicity. This is enforced by the recent post by Github 'analyzing' the copilot's code regurgitation, instead of retracting and retraining it on more defined subset of codebases. This thing is worthless at best, annoying at everything and terrifyingly capable of destroying every programmer's productivity at worst. But it will stay, it will grow because stupid execs will keep dreaming of replacing their engineering talent with this, and Microsoft will laugh all the way to the bank.
- WesolyKubeczek 5y ago
  Just until the software at the bank is rewritten using CoPilot and suddenly they can’t withdraw any money.
- beshrkayali 5y ago
  There's no way they didn't expect some backlash on this. So I think it's partly for the marketing gimmick. I'm sure there are people at Github tho who really think they're making something valuable unfortunately. Sadly, what they're not aware of is that they've become part of the experiment themselves. It's like expecting that an AI trained on Shakespeare novels would be able to "help" a writer write Shakespeare-like novels. Sure, they might get something that might fool some people, but are they a writer? I think software is a lot more like "writing" than it is like "building". What mostly annoys me is that this is a win-win for github regardless of the outcome. If people buy into it (even for just a while, and it currently seems like some really smart people are buying into it) they'll carve a huge piece of a new market. If it fails, they'll make it seem like an experiment into the whole ethical gray area of what should and shouldn't be used for training, and that they just wanted to draw attention to it.

tyingq 5y ago

I wish he would have tried to track down if the keys were in a public repo before asking Sendgrid about them. If they turned out to be only on Github private repos, that would be new and interesting info. Not saying putting keys in a private, but 3rd party hosted repo, is a terrific idea.

cabirum 5y ago

Can we please stop (mis)using the term "AI"? It just does not live up to most people's expectations. Copilot is a glorified Markov chain autocomplete sitting on a huge dump pile of data. It is not aware of constructs such as "licenses" or "secrets" most people would have expected from AI. To prevent it from spilling secrets everywhere, a developer ~~should teach the AI a concept of secrets and the meaning of licenses.~~ has to implement a filter. A regexp-based one will do, I guess.

armatav 5y ago
Yeah, this. Modern AI is not AI - AI is synthetic machine life and essentially always has been, in fiction and non-fictional idealism. Deep neural networks have stuck us in hope-fueled uncanny valley, and very smart people tend to become very confused about their technology when they're subject to it. This technology has its place about heuristic programming, definitely, but is not AI.
- TeMPOraL 5y ago
  Someone will definitely be quick to reply with the "AI is a moving goalpost, things stop being called AI when they work, for example..." - so I'll offer my counterpoint up front. These things were never AI except in marketing lingo and in connection to the research in machine learning. The common folk definition of AI doesn't change - it's still the same vision of computers in science fiction, with which you can converse, and which can think better than you (except in some specific ways in which are super-dumb - this is necessary for the story to have any plot).
  - armatav 5y ago
    Right. And just to be clear I love the field and most everybody in it, specifically for their idealism - I'm a practitioner myself - clearly everyone involved in the modern field of AI wants to leave a legacy of beneficial impact, and sees AI as their tool to do so. I just think if we're looking for life we won't find it in the gates of a transistor.
  - randallsquared 5y ago

fxtentacle 5y ago

If Copilot was trained only on public repos like they claim, then shouldn't those API keys already be disabled due to existing secret scanning tools? For example https://docs.github.com/en/code-security/secret-security/about-secret-scanning https://docs.github.com/en/code-security/secret-security/abo... The fact that Copilot recreates API keys that still work makes me wonder if they come from a semi-public place, because SendGrid is usually quite fast at blocking API keys that were accidentally made public.

sputknick 5y ago

I see this as a problem with the developers who are committing code, and not a problem with Copilot. if you make your secrets accessible then they might be accessed. Also if you are rotating your keys regularly that would also mitigate these issues. This is a problem with humans failing to execute known security best practices, not malicious AI doing something insidious.

speedgoose 5y ago

People put valid secrets in their public repository all the time. Just a quick search: https://grep.app/search?q=%28secret%7Capi%29_%3Fkey%5Cs%3A%3F%3D%20%5B%22%27%5D%5Ba-zA-Z0-9%5D%7B8%2C%7D%22&regexp=true https://grep.app/search?q=%28secret%7Capi%29_%3Fkey%5Cs%3A%3...

tgsovlerkhgsel 5y ago

Tweet taken down (?), does anyone have a mirror?

deleted 5y ago
[deleted]

mvolfik 5y ago

https://web.archive.org/web/20210705123028/https://twitter.com/alexjc/status/1411966249437995010 https://web.archive.org/web/20210705123028/https://twitter.c... > COPILOT SECURITY BREACH > SendGrid engineer reports API keys generated by the AI are not only valid but still functional. > GitHub CEO acknowledges the issue... still waiting for them to pull the plug or make a comment. :popcorn: Quoting https://twitter.com/pkell7/status/1411058236321681414 https://twitter.com/pkell7/status/1411058236321681414

ibraheemdev 5y ago

Was the tweet taken down?

Paradigma11 5y ago

I really dont think that stuff hosted in public repos can be classified as secrets.