8 ms·

If it was possible to point to where the code is actually stored in Copilot (i.e. run strings on the server and it spits out the copylefted code), I would be a

by fwlr 3y ago

If it was possible to point to where the code is actually stored in Copilot (i.e. run strings on the server and it spits out the copylefted code), I would be a lot more sympathetic to the view that LLMs are stealing code. Even if you decode the weights from floats to strings, you won't find the strings of code stored anywhere. It's probably not correct to say it's "learning" from code the same way humans are, but it is "learning" from code in the way LLMs learn. Fundamentally it seems to me that Copilot truly does synthesise code from some store of knowledge (even if it's hard to understand what this store of knowledge is), and the problem is that it's synthesising code identical to existing code. There are legal tools and also rhetoric that are designed for dealing with this problem of "synthesising something that is identical to an existing thing", and it's different tools and rhetoric from the ones we have for dealing with the problem of "stealing or copying existing things". It's valid to have an issue with with Copilot ingesting your code, but unfortunately people are largely using tools and rhetoric from the latter category to approach the issue, and that slight misapplication is causing their issues to fall on deaf ears a lot of the time.

pmontra 3y ago
What happens if I create a 10 lines function character by character identical to some proprietary or GPLled piece of code, without ever looking at that code nor knowing that it existed? I expect that the copyright holders could reach to me, tell me that my code is identical to theirs, believe or not that I independently created the same code, and at least ask me to stop distributing it. Of course t += total(e) is easier to defend than monthlyTotal += dailyTotal(expenses) because the chances that I didn't copy their code get slimmer and slimmer as the names of the identifiers and the structure of the code get more complex. If I actually looked at their code, it would be wiser from me to at least change the names, maybe also a little bit the structure of the code. That's the difference between being inspired by something and copying it. TL;DR: if Copilot generates a copy, it's a copy.
- gedy 3y ago
  I know the law doesn't work this way, but wonder if code that can be "learned" this easily should really be copyrightable, etc. These LLMs don't keep a copy of some code to paste in verbatim.
  - sanxiyn 3y ago
    LLMs totally do, and the fact can be confirmed by experiment. See "Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4": https://arxiv.org/abs/2305.00118 https://arxiv.org/abs/2305.00118
- lmm 3y ago
  > What happens if I create a 10 lines function character by character identical to some proprietary or GPLled piece of code, without ever looking at that code nor knowing that it existed? Then you're in the clear, though you may need to convince a court that that's what happened. (Patents and trademarks could still be issues, but there's no copyright issue)
goodpoint 3y ago
> it seems to me that Copilot truly does synthesise code from some store of knowledge It's a common mistake because we are not used to LLMs.
j16sdiz 3y ago
If I type out a copyrighted program character-by-character from my memory, does this somehow get exempted from copyright law?
- edgineer 3y ago
  Trying to be more precise: If the LLM reproduces a significant portion of a program token-by-token, is it a derivative work and is it not fair use?
  - telios 3y ago
    Now, I'm not a lawyer, so this isn't legal advice, but... A derivative work is not fair use. If you end up with a significant portion of another person's program in yours, (such that a substantial portion of your program is in some way related to their program), that will likely be a derivative work - but the definition of a derivative work depends on your jurisdiction and use-case. If you're unlawfully using the source material to produce a derivative work, you cannot copyright that derivative work under 17 U.S.C. § 103(a), and under the same section, you can only copyright your modifications, not the original. It would be hard to argue fair use in this case; fair use only really applies for parodies/criticism, reporting, and scholarly works - and generally that's an affirmative defense, rather than an express or implied right you have. Honestly, Copilot is difficult because Copilot can't be the author of the code; the person who used Copilot is the "author" of the code, and I think they'd be the ones liable for copyright infringement if copyrighted content ends up in their code. To argue someone performed copyright infringement, all you need is to prove (1) a valid copyright exists; (2) that the person had access to the work; (3) the person had the opportunity to steal the work; and (4) that protected elements of the work had been copied (afaik generally under a "substantially similar" standard). Copilot offers an easy way to check both (2) and (3) - a copyright holder could argue that people had access to their code through Copilot, and that Copilot offered an opportunity to steal the work.
dspillett 3y ago
> If it was possible to point to where the code is actually stored in Copilot (i.e. run strings on the server and it spits out the copylefted code), I would be a lot more sympathetic to the view that LLMs are stealing code. I think of it the other way around: I'd be more trusting that LLMs aren't a problem in that sense, if the big commercial entities linked to them used their own code to train them as well as public repositories. MS's codebase must contain much good training material, unless they think their own code is crap and are embarrassed to let the AI look at it! I am still undecided, but erring in the direction of not wanting my stuff to be used to train them, though I've long since passed Douglas Adam's "35 year's old" tech barrier so maybe I'm just not liking change… For now at least I won't be putting my own code in services like GitHub, then at least if someone else does or an LLM is trained on public sites generally, I've not explicitly agreed to anything that says the company can use my code that way - which you do (they say, I believe there is at least one court case brewing on the matter) when you sign up to the services (agreeing to their terms in the process) and use them that way.
- jwestbury 3y ago
  > MS's codebase must contain much good training material, unless they think their own code is crap and are embarrassed to let the AI look at it! Having worked at Microsoft, I'm not sure I want my coding assistant trained on Microsoft's codebase.
  - dspillett 3y ago
    But would it be worse overall than the quality in the public repositories that they did use for the training data?
NationOfJoe 3y ago
would you be able to elaborate on the, this isn't my field i would be interested to learn an alternative way to talk about all this. > legal tools and also rhetoric that are designed for dealing with this problem of "synthesising something that is identical to an existing thing"
- fwlr 3y ago
  Copyright infringement, intellectual property, patents, things like that.
wokwokwok 3y ago
Have I gone back in time? I feel like I’m reading the exact same sentence that someone posted about if StableDiffusion included images or was just “learning patterns” and totally generative from 6 to 8 months ago. People argued, oh so hard about it. “But stable diffusion is GB in size, how can it have embedded full images of a dataset that is so much larger? It’s not possible.” …but, it turns out it is possible (1) and it does, in fact, have embedded full copies of the source images. Now. Let’s talk about code… > and the problem is that it's synthesising code identical to existing code There’s a word for that. It’s called copying. Copying fragments, copying full text. It’s a black box, it spits out content that is indistinguishable (again, see stable diffusion, a lossy jpg of an image is the same image even when it is not bit-for-bit identical) from the input. That’s copying. The black box might be a neutral network, or a python script that reads a file from disk. The process is irrelevant. If you memorise a block of code and type it out by hand with no reference, you are copying it. These models copy code. These models have embedded full text copies of some code. What we do with that is an ethical question, but that it is true is not in dispute. It is true. It’s documented. If you think these models are not copying at least some training data as output you are factually, and provably incorrect. (1) - https://arstechnica.com/information-technology/2023/02/researchers-extract-training-images-from-stable-diffusion-but-its-difficult/ https://arstechnica.com/information-technology/2023/02/resea...
- casey2 3y ago
  It's clearly not a copy stop arguing in bad faith.
- deleted 3y ago
  [deleted]
- stavros 3y ago
  Does this storage argument also go for humans? If I ask a painter to paint paintings they've seen before, what will be their rate of copying the painting exactly?
  - wokwokwok 3y ago
    It doesn’t matter. The process is irrelevant. Black box. Input. Output. Is the output the same as some training input? It’s not rocket science. “…but humans…” argument is not relevant. This is not a human.
bombolo 3y ago
humans have learnt by looking at a few examples, and from there can code almost anything. Humans can also learn without examples, although it's harder. But I expect the 1st C programmers didn't have a K&R book to learn from. AI instead has looked at all the existing code and it has learnt to do no more than what the existing code can do. Do you see a difference?
- krisoft 3y ago
  > AI instead has looked at all the existing code and it has learnt to do no more than what the existing code can do. Are you sure about that?
  - bombolo 3y ago
    > Are you sure about that? Yes I'm quite sure. Unless you're claiming that source code existed before humans?
    - krisoft 3y ago
      I’m sorry but I don’t understand you. It sounds like what you are saying is that the AI cannot write code which would do something it hasn’t seen in the training set. That is how I interpreted the “it has learnt to do no more than what the existing code can do”. Do i understand you right? If so, how do you know that? Are you talking about the limitations of a particular implementation or a limitation of all AIs as a concept?
- stavros 3y ago
  No, because I can ask it for a piece of code that isn't in the training set and it will write it for me.
  - mikojan 3y ago
    You are missing the point. Humans make sense of the world with an extremely limited amount of information. I have not read all of GitHub. Comparatively, I have not read a fraction of it. And I could not read all of GitHub even if I wanted to. However, I do not need to read all of GitHub anyway because, as a human, I am capable of understanding. Current generation "AI" is not capable of understanding. Current generation "AI" merely computes the probability of any given word appearing next in this sentence based on an utterly absurd amount of raw data. That is not what humans do at all. We do not need this information. We cannot even process it.
klntsky 3y ago
> the problem is that it's synthesising code identical to existing code No, the problem is that it's illegal to do so.
pavlov 3y ago
> ‘There are legal tools and also rhetoric that are designed for dealing with this problem of "synthesising something that is identical to an existing thing", and it's different tools and rhetoric from the ones we have for dealing with the problem of "stealing or copying existing things".’ Copyright law applies in both cases. If you create work that’s substantially similar to an existing one, you’re risking copyright infringement even if none of the original work’s content was reproduced in the mechanical sense. If this were not the case, why would cover bands pay the original artists for rights to the songs? “I learned to play ‘Yesterday’ by heart” doesn’t mean you can do anything with the song without paying the Beatles. The same applies to machine learning, if all the model has learned is to imitate copyrighted works.
- fwlr 3y ago
  Yes, I do think that LLMs are probably doing copyright infringement. There is a long and rich heritage, in tech and elsewhere, of arguing that copyright infringement is not stealing, it is its own separate (though similar) crime and usually a lesser crime to some degree. Maybe my original point could be rephrased snarkily as “you may have a point but until you stop sounding like the RIAA calling everything ‘stealing’, a lot of people are going to ignore you”.
friendzis 3y ago
> If it was possible to point to where the code is actually stored in Copilot (i.e. run strings on the server and it spits out the copylefted code), I would be a lot more sympathetic to the view that LLMs are stealing code If it was possible to point where the pixel values are actually stored in a jpeg file (i.e. run dd on the file and it spits out the pixel values), I would be a lot more sympathetic to the view that LLMs are not stealing code.
- fwlr 3y ago
  In the very next sentence after the one you quote, I say “Even if you decode the weights from floats to strings, you won't find the strings of code stored anywhere.” If you decode the jpeg file you do get the pixel values.
  - friendzis 3y ago
    The whole debate is that when you decode the weights (i.e. "run AI on some prompt") you do in fact get training code reproduced verbatim. The fact that we do not have tools to analyze this decoding function analytically is orthogonal.
  - greysphere 3y ago
    You don't get the exact pixel values decoding a jpeg. But for a lot of uses, what you get is close enough. One could describe llms similarly.
wiml 3y ago
Every time I see this argument about LLMs I can't help but think of Borges' Pierre Menard, Author of the Quixote.
grrowl 3y ago
If we switch out LLMs for LZMA algorithms your reasoning still makes sense. Which is a real problem with your reasoning.
camgunz 3y ago
Aren't we just talking about a novel compression technique here. Like there was a voice codec Google released a while ago that essentially modeled your voice and then fed a piano roll of what you said through it. I don't really care whether there's a lossless, lossy, or AI compression of my stuff. I care that it's my stuff.
dingledork69 3y ago
How is any of that relevant? If i memorize your work and dissect my brain and run strings on the output you won't find anything either. Same happens if you put it in a .zip file. But both, and the language model, can spit it back out and infringe on your rights.

by fwlr 3y ago

pmontra 3y ago

What happens if I create a 10 lines function character by character identical to some proprietary or GPLled piece of code, without ever looking at that code nor knowing that it existed? I expect that the copyright holders could reach to me, tell me that my code is identical to theirs, believe or not that I independently created the same code, and at least ask me to stop distributing it. Of course t += total(e) is easier to defend than monthlyTotal += dailyTotal(expenses) because the chances that I didn't copy their code get slimmer and slimmer as the names of the identifiers and the structure of the code get more complex. If I actually looked at their code, it would be wiser from me to at least change the names, maybe also a little bit the structure of the code. That's the difference between being inspired by something and copying it. TL;DR: if Copilot generates a copy, it's a copy.

gedy 3y ago
I know the law doesn't work this way, but wonder if code that can be "learned" this easily should really be copyrightable, etc. These LLMs don't keep a copy of some code to paste in verbatim.
- sanxiyn 3y ago
  LLMs totally do, and the fact can be confirmed by experiment. See "Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4": https://arxiv.org/abs/2305.00118 https://arxiv.org/abs/2305.00118
lmm 3y ago
> What happens if I create a 10 lines function character by character identical to some proprietary or GPLled piece of code, without ever looking at that code nor knowing that it existed? Then you're in the clear, though you may need to convince a court that that's what happened. (Patents and trademarks could still be issues, but there's no copyright issue)

goodpoint 3y ago

> it seems to me that Copilot truly does synthesise code from some store of knowledge It's a common mistake because we are not used to LLMs.

j16sdiz 3y ago

If I type out a copyrighted program character-by-character from my memory, does this somehow get exempted from copyright law?

edgineer 3y ago
Trying to be more precise: If the LLM reproduces a significant portion of a program token-by-token, is it a derivative work and is it not fair use?
- telios 3y ago
  Now, I'm not a lawyer, so this isn't legal advice, but... A derivative work is not fair use. If you end up with a significant portion of another person's program in yours, (such that a substantial portion of your program is in some way related to their program), that will likely be a derivative work - but the definition of a derivative work depends on your jurisdiction and use-case. If you're unlawfully using the source material to produce a derivative work, you cannot copyright that derivative work under 17 U.S.C. § 103(a), and under the same section, you can only copyright your modifications, not the original. It would be hard to argue fair use in this case; fair use only really applies for parodies/criticism, reporting, and scholarly works - and generally that's an affirmative defense, rather than an express or implied right you have. Honestly, Copilot is difficult because Copilot can't be the author of the code; the person who used Copilot is the "author" of the code, and I think they'd be the ones liable for copyright infringement if copyrighted content ends up in their code. To argue someone performed copyright infringement, all you need is to prove (1) a valid copyright exists; (2) that the person had access to the work; (3) the person had the opportunity to steal the work; and (4) that protected elements of the work had been copied (afaik generally under a "substantially similar" standard). Copilot offers an easy way to check both (2) and (3) - a copyright holder could argue that people had access to their code through Copilot, and that Copilot offered an opportunity to steal the work.

dspillett 3y ago

> If it was possible to point to where the code is actually stored in Copilot (i.e. run strings on the server and it spits out the copylefted code), I would be a lot more sympathetic to the view that LLMs are stealing code. I think of it the other way around: I'd be more trusting that LLMs aren't a problem in that sense, if the big commercial entities linked to them used their own code to train them as well as public repositories. MS's codebase must contain much good training material, unless they think their own code is crap and are embarrassed to let the AI look at it! I am still undecided, but erring in the direction of not wanting my stuff to be used to train them, though I've long since passed Douglas Adam's "35 year's old" tech barrier so maybe I'm just not liking change… For now at least I won't be putting my own code in services like GitHub, then at least if someone else does or an LLM is trained on public sites generally, I've not explicitly agreed to anything that says the company can use my code that way - which you do (they say, I believe there is at least one court case brewing on the matter) when you sign up to the services (agreeing to their terms in the process) and use them that way.

jwestbury 3y ago
> MS's codebase must contain much good training material, unless they think their own code is crap and are embarrassed to let the AI look at it! Having worked at Microsoft, I'm not sure I want my coding assistant trained on Microsoft's codebase.
- dspillett 3y ago
  But would it be worse overall than the quality in the public repositories that they did use for the training data?

NationOfJoe 3y ago

would you be able to elaborate on the, this isn't my field i would be interested to learn an alternative way to talk about all this. > legal tools and also rhetoric that are designed for dealing with this problem of "synthesising something that is identical to an existing thing"

fwlr 3y ago
Copyright infringement, intellectual property, patents, things like that.

wokwokwok 3y ago

Have I gone back in time? I feel like I’m reading the exact same sentence that someone posted about if StableDiffusion included images or was just “learning patterns” and totally generative from 6 to 8 months ago. People argued, oh so hard about it. “But stable diffusion is GB in size, how can it have embedded full images of a dataset that is so much larger? It’s not possible.” …but, it turns out it is possible (1) and it does, in fact, have embedded full copies of the source images. Now. Let’s talk about code… > and the problem is that it's synthesising code identical to existing code There’s a word for that. It’s called copying. Copying fragments, copying full text. It’s a black box, it spits out content that is indistinguishable (again, see stable diffusion, a lossy jpg of an image is the same image even when it is not bit-for-bit identical) from the input. That’s copying. The black box might be a neutral network, or a python script that reads a file from disk. The process is irrelevant. If you memorise a block of code and type it out by hand with no reference, you are copying it. These models copy code. These models have embedded full text copies of some code. What we do with that is an ethical question, but that it is true is not in dispute. It is true. It’s documented. If you think these models are not copying at least some training data as output you are factually, and provably incorrect. (1) - https://arstechnica.com/information-technology/2023/02/researchers-extract-training-images-from-stable-diffusion-but-its-difficult/ https://arstechnica.com/information-technology/2023/02/resea...

casey2 3y ago
It's clearly not a copy stop arguing in bad faith.
deleted 3y ago
[deleted]
stavros 3y ago
Does this storage argument also go for humans? If I ask a painter to paint paintings they've seen before, what will be their rate of copying the painting exactly?
- wokwokwok 3y ago
  It doesn’t matter. The process is irrelevant. Black box. Input. Output. Is the output the same as some training input? It’s not rocket science. “…but humans…” argument is not relevant. This is not a human.

bombolo 3y ago

humans have learnt by looking at a few examples, and from there can code almost anything. Humans can also learn without examples, although it's harder. But I expect the 1st C programmers didn't have a K&R book to learn from. AI instead has looked at all the existing code and it has learnt to do no more than what the existing code can do. Do you see a difference?

krisoft 3y ago
> AI instead has looked at all the existing code and it has learnt to do no more than what the existing code can do. Are you sure about that?
- bombolo 3y ago
  > Are you sure about that? Yes I'm quite sure. Unless you're claiming that source code existed before humans?
  - krisoft 3y ago
    I’m sorry but I don’t understand you. It sounds like what you are saying is that the AI cannot write code which would do something it hasn’t seen in the training set. That is how I interpreted the “it has learnt to do no more than what the existing code can do”. Do i understand you right? If so, how do you know that? Are you talking about the limitations of a particular implementation or a limitation of all AIs as a concept?
stavros 3y ago
No, because I can ask it for a piece of code that isn't in the training set and it will write it for me.
- mikojan 3y ago
  You are missing the point. Humans make sense of the world with an extremely limited amount of information. I have not read all of GitHub. Comparatively, I have not read a fraction of it. And I could not read all of GitHub even if I wanted to. However, I do not need to read all of GitHub anyway because, as a human, I am capable of understanding. Current generation "AI" is not capable of understanding. Current generation "AI" merely computes the probability of any given word appearing next in this sentence based on an utterly absurd amount of raw data. That is not what humans do at all. We do not need this information. We cannot even process it.

klntsky 3y ago

> the problem is that it's synthesising code identical to existing code No, the problem is that it's illegal to do so.

pavlov 3y ago

> ‘There are legal tools and also rhetoric that are designed for dealing with this problem of "synthesising something that is identical to an existing thing", and it's different tools and rhetoric from the ones we have for dealing with the problem of "stealing or copying existing things".’ Copyright law applies in both cases. If you create work that’s substantially similar to an existing one, you’re risking copyright infringement even if none of the original work’s content was reproduced in the mechanical sense. If this were not the case, why would cover bands pay the original artists for rights to the songs? “I learned to play ‘Yesterday’ by heart” doesn’t mean you can do anything with the song without paying the Beatles. The same applies to machine learning, if all the model has learned is to imitate copyrighted works.

fwlr 3y ago
Yes, I do think that LLMs are probably doing copyright infringement. There is a long and rich heritage, in tech and elsewhere, of arguing that copyright infringement is not stealing, it is its own separate (though similar) crime and usually a lesser crime to some degree. Maybe my original point could be rephrased snarkily as “you may have a point but until you stop sounding like the RIAA calling everything ‘stealing’, a lot of people are going to ignore you”.

friendzis 3y ago

> If it was possible to point to where the code is actually stored in Copilot (i.e. run strings on the server and it spits out the copylefted code), I would be a lot more sympathetic to the view that LLMs are stealing code If it was possible to point where the pixel values are actually stored in a jpeg file (i.e. run dd on the file and it spits out the pixel values), I would be a lot more sympathetic to the view that LLMs are not stealing code.

fwlr 3y ago
In the very next sentence after the one you quote, I say “Even if you decode the weights from floats to strings, you won't find the strings of code stored anywhere.” If you decode the jpeg file you do get the pixel values.
- friendzis 3y ago
  The whole debate is that when you decode the weights (i.e. "run AI on some prompt") you do in fact get training code reproduced verbatim. The fact that we do not have tools to analyze this decoding function analytically is orthogonal.
- greysphere 3y ago
  You don't get the exact pixel values decoding a jpeg. But for a lot of uses, what you get is close enough. One could describe llms similarly.

wiml 3y ago

Every time I see this argument about LLMs I can't help but think of Borges' Pierre Menard, Author of the Quixote.

grrowl 3y ago

If we switch out LLMs for LZMA algorithms your reasoning still makes sense. Which is a real problem with your reasoning.

camgunz 3y ago

Aren't we just talking about a novel compression technique here. Like there was a voice codec Google released a while ago that essentially modeled your voice and then fed a piano roll of what you said through it. I don't really care whether there's a lossless, lossy, or AI compression of my stuff. I care that it's my stuff.

dingledork69 3y ago

How is any of that relevant? If i memorize your work and dissect my brain and run strings on the output you won't find anything either. Same happens if you put it in a .zip file. But both, and the language model, can spit it back out and infringe on your rights.