7 ms·

The claim for AI systems like this is that it has actually learned something and is generating code from scratch. Oftentimes the authors will claim regurgitatio

The claim for AI systems like this is that it has actually learned something and is generating code from scratch. Oftentimes the authors will claim regurgitation is simply not possible, and this example shows that's a lie. Many arguments on the benefits, legality and power of AI systems rely on this claim. To turn around now and say it's OK to regurgitate in the right setting is to move the goalposts.

caconym_ 5y ago
> Oftentimes the authors will claim regurgitation is simply not possible Do the Copilot authors claim this? I get that you're suggesting that Copilot may benefit from absolute claims made by the authors of other, similar systems (or their proponents), but I also don't think it's reasonable to exclude nuance and the specifics of Copilot from ongoing discussions on that basis. The Copilot authors have publicly acknowledged the regurgitation problem, and by their account are working on solutions to it (e.g. attribution at suggestion-generation time) that don't involve sweeping it under the rug.
- rckstr 5y ago
  They did! In the faq which I can't find anymore they said: >GitHub Copilot is a code synthesizer, not a search engine: the vast majority of the code that it suggests is uniquely generated and has never been seen before. We found that about 0.1% of the time, the suggestion may contain some snippets that are verbatim from the training set.
  - caconym_ 5y ago
    This actually seems like an explicit acknowledgement that regurgitation is possible, and not remotely a claim that it is "simply not possible". It stands to reason that cases where people are intentionally trying to produce regurgitation will strongly overlap with the minority of cases where it actually happens. So I think we are probably suffering from some selection bias in discussions on HN and similar forums--that might be unavoidable, and it certainly stimulates some interesting discussion, but we should try to avoid misrepresenting the product as a whole and/or what its creators have said about it.
    - sombremesa 5y ago
      I think only Github's lawyers would interpret what GP posted the way you did. Looks like weasel wording to make such an interpretation possible, while making customers believe that code is more or less synthesized in realtime. "Snippets" makes one think one or two lines of code, not entire functions and classes.
      - caconym_ 5y ago
        I think that until somebody shows that Copilot is willing to copy distinctive code fragments verbatim, unprompted, with a high occurrence rate, I'm not going to start accusing Github of building an engine to cynically exploit the IP rights of open source copyright holders for profit. I've seen no evidence of that, and in absence of evidence I prefer to remain neutral and open-minded. How would that work, anyway? Rare, distinctive code forms seem much more difficult for an ML thing to suggest with a high-ish confidence level, since there won't be much training data. The Quake thing makes sense because it's one of the most famous sections of code in the world, and probably exists in thousands of places in the public Github corpus. I'm emphasizing distinctive because a lot of boilerplate takes up a lot of room, but still doesn't make a reasonable argument for copyright infringement when yours looks like somebody else's.
        sombremesa 5y ago
        It looks like you're responding to the wrong comment. I don't recall alleging that Github is "building an engine to cynically exploit the IP rights of open source copyright holders for profit".
        caconym_ 5y ago
        > I think only Github's lawyers would interpret what GP posted the way you did. Looks like weasel wording to make such an interpretation possible, So what are you suggesting here, except that Github is attempting a legal sleight-of-hand to hide real infringement? > while making customers believe that code is more or less synthesized in realtime. What are you suggesting here except that Github is (essentially) lying to customers, making them believe something that is substantially untrue? When I say "building an engine to cynically exploit the IP rights of open source copyright holders for profit", I am talking about a scenario in which they are sweeping legitimate IP concerns under the rug with bad faith legal weaselry and misrepresentation of how the product functions, etc., to chase profit. I do not see how that is substantially different from the implications of your comment, especially in the context of this subthread. Could you enlighten me as to how your intended meaning substantially differs from my interpretation? If you don't mean to accuse Github of malfeasance, we probably don't have much to discuss.
  - YeGoblynQueenne 5y ago
    There's a risk of confirmation bias though, because the search was performed by the developers of the system who are strongly motivated to find no problem with their work. I have done this to myself many, many times. I look, carefully, at length, for problems with my work until I satisfy myself that there's no obvious problem with it. Then someone else points out the obvious problem I was overlooking. Actually, this has happened often enough and it's painful enough that I learned to really look nowadays. In the context of the search for "snippets that are verbatim from the training set" there's all sorts of things that can go wrong. The search (a regex search I think?) can be unintentionally made too weak to catch obvious cases. Or too strong, probably. The search for "snippets that are verbatim from the training set" may ignore snippets that are 80% verbatim. Or the code generated during the experiment can be generated in such a way as to only generate verbatim snippets very rarely, contrary to more typical use. And so on and so forth. There's so many ways to fool oneself when looking for errors in one's own work. Edit: They explain their search methodology and with only a quick look I gave it, it seems legit, but it was a quick look. The devil is in the details, yes? Maybe people who are really interested in this issue should take a closer look.
- ohazi 5y ago
  Nat Friedman explicitly stated that it shouldn't regurgitate [0]: > It shouldn't do that, and we are taking steps to avoid reciting training data in the output He's being woefully naive. To put it bluntly, we don't know how to build a neural network that isn't capable of spitting out training data. The techniques he pointed to in other threads are academic experiments, and nobody seems to have a credible explanation for why we should believe that they work. [0] https://news.ycombinator.com/item?id=27677177 https://news.ycombinator.com/item?id=27677177
  - caconym_ 5y ago
    "Shouldn't" isn't the same as "doesn't". I'm not anything close to an ML expert, and I have no opinion on whether what they're aiming for is possible, but this document^[1] (linked in your linked comment) states explicitly that they are aware of the recitation issue and are taking steps to mitigate it. So, in the context of the comment I replied to, I think Github is very far from claiming that recitation is "simply not possible". ^[1] https://docs.github.com/en/github/copilot/research-recitation https://docs.github.com/en/github/copilot/research-recitatio...
    - ohazi 5y ago
      That kind of bullshit phrasing can only get you so far. It's like if some corporate PR department told you "we're aware of the halting problem, and are taking steps to mitigate it." You would rightly laugh them out of the room. It's not going to work, and the people making these statements either don't understand how much they don't understand, or are deluding themselves, or are actively lying to us. An honest answer would be something like "We are aware that this is a problem, and solving it is an active area of research for us, and for the machine learning community at large. While we believe that we will eventually be able to mitigate the problem to an acceptable degree, it is not yet known whether this category of problem can be fully solved."
      - caconym_ 5y ago
        You're using some pretty strong language here, but do you have any more substantive criticisms of the analysis they present at https://docs.github.com/en/github/copilot/research-recitation https://docs.github.com/en/github/copilot/research-recitatio... ? They seem to think the incidence of meaningful (i.e. substantively infringing) recitation is very low, and that their solution in those cases will be attribution rather than elimination. Again, I'm not an ML expert, but that sounds a lot more reasonable to me than announcing one's intention to solve the halting problem.

caconym_ 5y ago

> Oftentimes the authors will claim regurgitation is simply not possible Do the Copilot authors claim this? I get that you're suggesting that Copilot may benefit from absolute claims made by the authors of other, similar systems (or their proponents), but I also don't think it's reasonable to exclude nuance and the specifics of Copilot from ongoing discussions on that basis. The Copilot authors have publicly acknowledged the regurgitation problem, and by their account are working on solutions to it (e.g. attribution at suggestion-generation time) that don't involve sweeping it under the rug.

rckstr 5y ago
They did! In the faq which I can't find anymore they said: >GitHub Copilot is a code synthesizer, not a search engine: the vast majority of the code that it suggests is uniquely generated and has never been seen before. We found that about 0.1% of the time, the suggestion may contain some snippets that are verbatim from the training set.
- caconym_ 5y ago
  This actually seems like an explicit acknowledgement that regurgitation is possible, and not remotely a claim that it is "simply not possible". It stands to reason that cases where people are intentionally trying to produce regurgitation will strongly overlap with the minority of cases where it actually happens. So I think we are probably suffering from some selection bias in discussions on HN and similar forums--that might be unavoidable, and it certainly stimulates some interesting discussion, but we should try to avoid misrepresenting the product as a whole and/or what its creators have said about it.
  - sombremesa 5y ago
    I think only Github's lawyers would interpret what GP posted the way you did. Looks like weasel wording to make such an interpretation possible, while making customers believe that code is more or less synthesized in realtime. "Snippets" makes one think one or two lines of code, not entire functions and classes.
    - caconym_ 5y ago
      I think that until somebody shows that Copilot is willing to copy distinctive code fragments verbatim, unprompted, with a high occurrence rate, I'm not going to start accusing Github of building an engine to cynically exploit the IP rights of open source copyright holders for profit. I've seen no evidence of that, and in absence of evidence I prefer to remain neutral and open-minded. How would that work, anyway? Rare, distinctive code forms seem much more difficult for an ML thing to suggest with a high-ish confidence level, since there won't be much training data. The Quake thing makes sense because it's one of the most famous sections of code in the world, and probably exists in thousands of places in the public Github corpus. I'm emphasizing distinctive because a lot of boilerplate takes up a lot of room, but still doesn't make a reasonable argument for copyright infringement when yours looks like somebody else's.
      - sombremesa 5y ago
        It looks like you're responding to the wrong comment. I don't recall alleging that Github is "building an engine to cynically exploit the IP rights of open source copyright holders for profit".
        caconym_ 5y ago
        > I think only Github's lawyers would interpret what GP posted the way you did. Looks like weasel wording to make such an interpretation possible, So what are you suggesting here, except that Github is attempting a legal sleight-of-hand to hide real infringement? > while making customers believe that code is more or less synthesized in realtime. What are you suggesting here except that Github is (essentially) lying to customers, making them believe something that is substantially untrue? When I say "building an engine to cynically exploit the IP rights of open source copyright holders for profit", I am talking about a scenario in which they are sweeping legitimate IP concerns under the rug with bad faith legal weaselry and misrepresentation of how the product functions, etc., to chase profit. I do not see how that is substantially different from the implications of your comment, especially in the context of this subthread. Could you enlighten me as to how your intended meaning substantially differs from my interpretation? If you don't mean to accuse Github of malfeasance, we probably don't have much to discuss.
- YeGoblynQueenne 5y ago
  There's a risk of confirmation bias though, because the search was performed by the developers of the system who are strongly motivated to find no problem with their work. I have done this to myself many, many times. I look, carefully, at length, for problems with my work until I satisfy myself that there's no obvious problem with it. Then someone else points out the obvious problem I was overlooking. Actually, this has happened often enough and it's painful enough that I learned to really look nowadays. In the context of the search for "snippets that are verbatim from the training set" there's all sorts of things that can go wrong. The search (a regex search I think?) can be unintentionally made too weak to catch obvious cases. Or too strong, probably. The search for "snippets that are verbatim from the training set" may ignore snippets that are 80% verbatim. Or the code generated during the experiment can be generated in such a way as to only generate verbatim snippets very rarely, contrary to more typical use. And so on and so forth. There's so many ways to fool oneself when looking for errors in one's own work. Edit: They explain their search methodology and with only a quick look I gave it, it seems legit, but it was a quick look. The devil is in the details, yes? Maybe people who are really interested in this issue should take a closer look.
ohazi 5y ago
Nat Friedman explicitly stated that it shouldn't regurgitate [0]: > It shouldn't do that, and we are taking steps to avoid reciting training data in the output He's being woefully naive. To put it bluntly, we don't know how to build a neural network that isn't capable of spitting out training data. The techniques he pointed to in other threads are academic experiments, and nobody seems to have a credible explanation for why we should believe that they work. [0] https://news.ycombinator.com/item?id=27677177 https://news.ycombinator.com/item?id=27677177
- caconym_ 5y ago
  "Shouldn't" isn't the same as "doesn't". I'm not anything close to an ML expert, and I have no opinion on whether what they're aiming for is possible, but this document^[1] (linked in your linked comment) states explicitly that they are aware of the recitation issue and are taking steps to mitigate it. So, in the context of the comment I replied to, I think Github is very far from claiming that recitation is "simply not possible". ^[1] https://docs.github.com/en/github/copilot/research-recitation https://docs.github.com/en/github/copilot/research-recitatio...
  - ohazi 5y ago
    That kind of bullshit phrasing can only get you so far. It's like if some corporate PR department told you "we're aware of the halting problem, and are taking steps to mitigate it." You would rightly laugh them out of the room. It's not going to work, and the people making these statements either don't understand how much they don't understand, or are deluding themselves, or are actively lying to us. An honest answer would be something like "We are aware that this is a problem, and solving it is an active area of research for us, and for the machine learning community at large. While we believe that we will eventually be able to mitigate the problem to an acceptable degree, it is not yet known whether this category of problem can be fully solved."
    - caconym_ 5y ago
      You're using some pretty strong language here, but do you have any more substantive criticisms of the analysis they present at https://docs.github.com/en/github/copilot/research-recitation https://docs.github.com/en/github/copilot/research-recitatio... ? They seem to think the incidence of meaningful (i.e. substantively infringing) recitation is very low, and that their solution in those cases will be attribution rather than elimination. Again, I'm not an ML expert, but that sounds a lot more reasonable to me than announcing one's intention to solve the halting problem.