4 ms·

Can you cite any information on this not being possible for the vast majority? Or is it simply that the correct prompt hasn't been written for all possible cas

by tarnith 2mo ago

Can you cite any information on this not being possible for the vast majority? Or is it simply that the correct prompt hasn't been written for all possible cases? I also fail to see the difference if logic/harnessing is added around a vector database that can output the complete corpus, but simply is instructed not to. It very clearly is still compressing the information into the vector weights, and then recovering that information, thus the information is encoded. Why is a vector database somehow completely different from maintaining a library of the text itself?

satvikpendem 2mo ago
Information entropy. The amount of data an LLM ingests cannot be compressed to the size of the weights even at maximum theoretical compression.
- insanitybit 2mo ago
  Is that relevant? I can use a lossy compression algorithm such that the original could never be recovered from the image I've produced, but that derived image would surely be under copyright. LLMs are obviously capable of producing "exact" phrases as well. Ask it to give you famous quotes, it can do it. Ask it to read a paper for you and cite it, it can do it.
  - flir 2mo ago
    > but that derived image would surely be under copyright. I wouldn't bet on that. https://en.wikipedia.org/wiki/Campbell%27s_Soup_Cans https://en.wikipedia.org/wiki/Campbell%27s_Soup_Cans
    - insanitybit 2mo ago
      I don't know that this really challenges anything relating to compression.
      - flir 2mo ago
        Ok, reductio ad absurdum. Here's a highly compressed representation of The Lord of The Rings (all three volumes): 1 Obviously, fidelity when uncompressing it is not great, but I can assure you it was lossily compressed from the original text. Is it infringing the original's copyright? I have to assume you'd agree that the answer is "no". If I had compressed it by removing the letters x y and z, I'd agree with you that my "compressed" version is infringing. So what we've got here is a spectrum with two ridiculous extremes, and a question: When has the artifact been compressed so heavily that it no longer infringes the copyright of the original? I suggest "irretrievability" is a pretty good threshold for that question. Otherwise you're into "we know it infringes our copyright. Don't ask us to prove it, we just know it, ok?" Given the sheer volume of text that an LLM gets trained on, and how small the output is, it seems obvious that 99% of it can no longer be recovered - the process is "lossy" to the point of irretrievability, and only a statistical smear is left behind. That's why I think only the copyright claims that can show infringement in court (Harpy Potter, et al.) have merit. And a court will still have to decide "how much is too much" but at least there's case law for that. (Incidentally, I compressed the Mona Lisa to a single pixel. It was #3D3526).
        insanitybit 2mo ago
        My entire point is that compression is irrelevant. A lossily compressed image can absolutely still be in breach of the original's copyright. Showing that some kind of compressed artifact may not be doesn't change that.
        flir 2mo ago
        "I can use a lossy compression algorithm such that the original could never be recovered from the image I've produced, but that derived image would surely be under copyright." I've tried my best to show where I think you're wrong. I think all that's left is arguing over the exact definitions of "recoverable" and "irretrievable". As I said, the courts will have to decide that.
        2mo ago
  - satvikpendem 2mo ago
    Why would a derived image be under copyright?
    - insanitybit 2mo ago
      Because that's legally the case? I don't understand the question. Using a lossy compression algorithm on an image does not remove its copyright protection.
      - satvikpendem 2mo ago
        Taking in an image then creating something new based on the ideas of that image is legally permissible, if you mean that you think AI training produces only derivative works.
        insanitybit 2mo ago
        I am not sure what you're trying to say. Compressing an image is creating a derived work, it is still subject to the copyright of the original.
        satvikpendem 2mo ago
        What AI is doing is not analogous to compression so even if you think they are they same, it legally and technically speaking is not subject to the copyright of the original.
        insanitybit 2mo ago
        Are you missing the comment that I'd responded to? > Information entropy. The amount of data an LLM ingests cannot be compressed to the size of the weights even at maximum theoretical compression.
        satvikpendem 2mo ago
consensus1 2mo ago
You are asking to prove a negative. But even assuming that the model is capable of returning every bit of its training data verbatim (a mathematical impossibility) that would not be enough as mere capability is insufficient here. If capability alone were the standard any library that also has a photocopier / scanner would be in violation. To prove distribution of copyrighted materials it would have to be practical and actually used in the wild by people to circumvent copyright and generate copies of those works. Again, I can't prove a negative, but that isn't the standard, and nobody has shown a practical exploit here.
- flir 2mo ago
  There was a paper a while back where (from memory) they managed to coax 75% of the original text of some internet-popular books out of an LLM. Harry Potter, 1984, etc. That's why I said it was possible for some texts. My assumption is that multiple copies in the training data "wear a deeper groove". I believe those are infringing, and should be dealt with on a case-by-case basis. But the vast majority of text doesn't wear that groove. (Edit: Think it was this one https://arxiv.org/abs/2601.02671 https://arxiv.org/abs/2601.02671)
flir 2mo ago
Nah, I can't prove a negative. But Common Crawl is 12 petabytes and is not the largest part of what these models get trained on. DeepSeek v4 Pro is, what, 865GB? That's one hell of a compression ratio, if it can do what you claim.

by tarnith 2mo ago

satvikpendem 2mo ago

Information entropy. The amount of data an LLM ingests cannot be compressed to the size of the weights even at maximum theoretical compression.

insanitybit 2mo ago
Is that relevant? I can use a lossy compression algorithm such that the original could never be recovered from the image I've produced, but that derived image would surely be under copyright. LLMs are obviously capable of producing "exact" phrases as well. Ask it to give you famous quotes, it can do it. Ask it to read a paper for you and cite it, it can do it.
- flir 2mo ago
  > but that derived image would surely be under copyright. I wouldn't bet on that. https://en.wikipedia.org/wiki/Campbell%27s_Soup_Cans https://en.wikipedia.org/wiki/Campbell%27s_Soup_Cans
  - insanitybit 2mo ago
    I don't know that this really challenges anything relating to compression.
    - flir 2mo ago
      Ok, reductio ad absurdum. Here's a highly compressed representation of The Lord of The Rings (all three volumes): 1 Obviously, fidelity when uncompressing it is not great, but I can assure you it was lossily compressed from the original text. Is it infringing the original's copyright? I have to assume you'd agree that the answer is "no". If I had compressed it by removing the letters x y and z, I'd agree with you that my "compressed" version is infringing. So what we've got here is a spectrum with two ridiculous extremes, and a question: When has the artifact been compressed so heavily that it no longer infringes the copyright of the original? I suggest "irretrievability" is a pretty good threshold for that question. Otherwise you're into "we know it infringes our copyright. Don't ask us to prove it, we just know it, ok?" Given the sheer volume of text that an LLM gets trained on, and how small the output is, it seems obvious that 99% of it can no longer be recovered - the process is "lossy" to the point of irretrievability, and only a statistical smear is left behind. That's why I think only the copyright claims that can show infringement in court (Harpy Potter, et al.) have merit. And a court will still have to decide "how much is too much" but at least there's case law for that. (Incidentally, I compressed the Mona Lisa to a single pixel. It was #3D3526).
      - insanitybit 2mo ago
        My entire point is that compression is irrelevant. A lossily compressed image can absolutely still be in breach of the original's copyright. Showing that some kind of compressed artifact may not be doesn't change that.
        flir 2mo ago
        "I can use a lossy compression algorithm such that the original could never be recovered from the image I've produced, but that derived image would surely be under copyright." I've tried my best to show where I think you're wrong. I think all that's left is arguing over the exact definitions of "recoverable" and "irretrievable". As I said, the courts will have to decide that.
        2mo ago
- satvikpendem 2mo ago
  Why would a derived image be under copyright?
  - insanitybit 2mo ago
    Because that's legally the case? I don't understand the question. Using a lossy compression algorithm on an image does not remove its copyright protection.
    - satvikpendem 2mo ago
      Taking in an image then creating something new based on the ideas of that image is legally permissible, if you mean that you think AI training produces only derivative works.
      - insanitybit 2mo ago
        I am not sure what you're trying to say. Compressing an image is creating a derived work, it is still subject to the copyright of the original.
        satvikpendem 2mo ago
        What AI is doing is not analogous to compression so even if you think they are they same, it legally and technically speaking is not subject to the copyright of the original.
        insanitybit 2mo ago
        Are you missing the comment that I'd responded to? > Information entropy. The amount of data an LLM ingests cannot be compressed to the size of the weights even at maximum theoretical compression.
        satvikpendem 2mo ago

consensus1 2mo ago

You are asking to prove a negative. But even assuming that the model is capable of returning every bit of its training data verbatim (a mathematical impossibility) that would not be enough as mere capability is insufficient here. If capability alone were the standard any library that also has a photocopier / scanner would be in violation. To prove distribution of copyrighted materials it would have to be practical and actually used in the wild by people to circumvent copyright and generate copies of those works. Again, I can't prove a negative, but that isn't the standard, and nobody has shown a practical exploit here.

flir 2mo ago
There was a paper a while back where (from memory) they managed to coax 75% of the original text of some internet-popular books out of an LLM. Harry Potter, 1984, etc. That's why I said it was possible for some texts. My assumption is that multiple copies in the training data "wear a deeper groove". I believe those are infringing, and should be dealt with on a case-by-case basis. But the vast majority of text doesn't wear that groove. (Edit: Think it was this one https://arxiv.org/abs/2601.02671 https://arxiv.org/abs/2601.02671)

flir 2mo ago

Nah, I can't prove a negative. But Common Crawl is 12 petabytes and is not the largest part of what these models get trained on. DeepSeek v4 Pro is, what, 865GB? That's one hell of a compression ratio, if it can do what you claim.