3 ms·

Nat Friedman explicitly stated that it shouldn't regurgitate [0]: > It shouldn't do that, and we are taking steps to avoid reciting training data in the output

by ohazi 5y ago

Nat Friedman explicitly stated that it shouldn't regurgitate [0]: > It shouldn't do that, and we are taking steps to avoid reciting training data in the output He's being woefully naive. To put it bluntly, we don't know how to build a neural network that isn't capable of spitting out training data. The techniques he pointed to in other threads are academic experiments, and nobody seems to have a credible explanation for why we should believe that they work. [0] https://news.ycombinator.com/item?id=27677177 https://news.ycombinator.com/item?id=27677177

caconym_ 5y ago
"Shouldn't" isn't the same as "doesn't". I'm not anything close to an ML expert, and I have no opinion on whether what they're aiming for is possible, but this document^[1] (linked in your linked comment) states explicitly that they are aware of the recitation issue and are taking steps to mitigate it. So, in the context of the comment I replied to, I think Github is very far from claiming that recitation is "simply not possible". ^[1] https://docs.github.com/en/github/copilot/research-recitation https://docs.github.com/en/github/copilot/research-recitatio...
- ohazi 5y ago
  That kind of bullshit phrasing can only get you so far. It's like if some corporate PR department told you "we're aware of the halting problem, and are taking steps to mitigate it." You would rightly laugh them out of the room. It's not going to work, and the people making these statements either don't understand how much they don't understand, or are deluding themselves, or are actively lying to us. An honest answer would be something like "We are aware that this is a problem, and solving it is an active area of research for us, and for the machine learning community at large. While we believe that we will eventually be able to mitigate the problem to an acceptable degree, it is not yet known whether this category of problem can be fully solved."
  - caconym_ 5y ago
    You're using some pretty strong language here, but do you have any more substantive criticisms of the analysis they present at https://docs.github.com/en/github/copilot/research-recitation https://docs.github.com/en/github/copilot/research-recitatio... ? They seem to think the incidence of meaningful (i.e. substantively infringing) recitation is very low, and that their solution in those cases will be attribution rather than elimination. Again, I'm not an ML expert, but that sounds a lot more reasonable to me than announcing one's intention to solve the halting problem.
    - csande17 5y ago
      The biggest issue with that analysis is that their model is clearly very able to copy code and change the variable names, copying code and changing variable names is very clearly still "copying", and the analysis doesn't seem to include that in its definition of "recitation event".
      - caconym_ 5y ago
        I'd fully expect it to copy code and change variable names in a lot of cases--if it wants to achieve the goal of filling in boilerplate, how could it do anything else? That's pretty much the definition of boilerplate: it's largely the same every time you write it. What's less clear to me is that Copilot regularly does that sort of thing with code distinctive enough that it could reasonably be said to constitute copyright infringement. If somebody's actually shown that it does, I'd love to see that analysis.
    - ohazi 5y ago
      They had some people use the thing for a while, and concluded "Hey look, it doesn't seem to quote verbatim very often. Yay!" There is nothing in there that describes any sort of mitigation. The three sentences about an attribution search at the very end are aspirational at best, and are presented as "obvious" even though it's not at all clear that such a fuzzy search can be implemented reliably. I use the halting problem as an analogy because their naive attempts to address this problem feel a lot like naive attempts to get around the halting problem ("just do a quick search for anything that looks like a loop," "just have a big list of valid programs," etc.). I can perform a similar analysis of programs that I run in my terminal and come to a similar "Hey look, most of them halt! Yay!" conclusion. I can spin a story about how most of the ones that don't halt are doing so intentionally because they're daemons. But this approach is inherently flawed. I can use a fuzz tester to come up with an infinite number of inputs that cause something as simple as 'ls' to run forever. Similarly, I can come up with an infinite number of adversarial inputs that attempt to make Copilot spit out training data. Some of them will work. Some of them will produce something that's close enough to training data to be a concern, but that their "attribution search" will fail to catch. That's the "open research question" that they need to solve. We don't have a general solution to this problem yet, and we may never have one. They're trying to pass off a hand-wavey "we can implement some rules and it won't be a problem most of the time" solution as adequate. I don't see any reason to believe that it will be adequate. Every attempt I've seen at using logic to try and coax a machine learning model into not behaving pathologically around edge cases has fallen flat on its face.

by ohazi 5y ago

caconym_ 5y ago

"Shouldn't" isn't the same as "doesn't". I'm not anything close to an ML expert, and I have no opinion on whether what they're aiming for is possible, but this document^[1] (linked in your linked comment) states explicitly that they are aware of the recitation issue and are taking steps to mitigate it. So, in the context of the comment I replied to, I think Github is very far from claiming that recitation is "simply not possible". ^[1] https://docs.github.com/en/github/copilot/research-recitation https://docs.github.com/en/github/copilot/research-recitatio...

ohazi 5y ago
That kind of bullshit phrasing can only get you so far. It's like if some corporate PR department told you "we're aware of the halting problem, and are taking steps to mitigate it." You would rightly laugh them out of the room. It's not going to work, and the people making these statements either don't understand how much they don't understand, or are deluding themselves, or are actively lying to us. An honest answer would be something like "We are aware that this is a problem, and solving it is an active area of research for us, and for the machine learning community at large. While we believe that we will eventually be able to mitigate the problem to an acceptable degree, it is not yet known whether this category of problem can be fully solved."
- caconym_ 5y ago
  You're using some pretty strong language here, but do you have any more substantive criticisms of the analysis they present at https://docs.github.com/en/github/copilot/research-recitation https://docs.github.com/en/github/copilot/research-recitatio... ? They seem to think the incidence of meaningful (i.e. substantively infringing) recitation is very low, and that their solution in those cases will be attribution rather than elimination. Again, I'm not an ML expert, but that sounds a lot more reasonable to me than announcing one's intention to solve the halting problem.
  - csande17 5y ago
    The biggest issue with that analysis is that their model is clearly very able to copy code and change the variable names, copying code and changing variable names is very clearly still "copying", and the analysis doesn't seem to include that in its definition of "recitation event".
    - caconym_ 5y ago
      I'd fully expect it to copy code and change variable names in a lot of cases--if it wants to achieve the goal of filling in boilerplate, how could it do anything else? That's pretty much the definition of boilerplate: it's largely the same every time you write it. What's less clear to me is that Copilot regularly does that sort of thing with code distinctive enough that it could reasonably be said to constitute copyright infringement. If somebody's actually shown that it does, I'd love to see that analysis.
  - ohazi 5y ago
    They had some people use the thing for a while, and concluded "Hey look, it doesn't seem to quote verbatim very often. Yay!" There is nothing in there that describes any sort of mitigation. The three sentences about an attribution search at the very end are aspirational at best, and are presented as "obvious" even though it's not at all clear that such a fuzzy search can be implemented reliably. I use the halting problem as an analogy because their naive attempts to address this problem feel a lot like naive attempts to get around the halting problem ("just do a quick search for anything that looks like a loop," "just have a big list of valid programs," etc.). I can perform a similar analysis of programs that I run in my terminal and come to a similar "Hey look, most of them halt! Yay!" conclusion. I can spin a story about how most of the ones that don't halt are doing so intentionally because they're daemons. But this approach is inherently flawed. I can use a fuzz tester to come up with an infinite number of inputs that cause something as simple as 'ls' to run forever. Similarly, I can come up with an infinite number of adversarial inputs that attempt to make Copilot spit out training data. Some of them will work. Some of them will produce something that's close enough to training data to be a concern, but that their "attribution search" will fail to catch. That's the "open research question" that they need to solve. We don't have a general solution to this problem yet, and we may never have one. They're trying to pass off a hand-wavey "we can implement some rules and it won't be a problem most of the time" solution as adequate. I don't see any reason to believe that it will be adequate. Every attempt I've seen at using logic to try and coax a machine learning model into not behaving pathologically around edge cases has fallen flat on its face.