3 ms·

> Unless I misunderstand and the “process[ing] for language modeling” is an entirely irreversible process. In the case of The Pile, "processing for language mo

> Unless I misunderstand and the “process[ing] for language modeling” is an entirely irreversible process. In the case of The Pile, "processing for language modelling" means "converting epub and pdf into plain text, maybe deduplicating, maybe removing some sorts of detectably malformed files" So not a particularly lossy conversion.

layer8 3y ago
I see, thanks. Yes, in that case, I don’t see how this can possibly not constitute copyright infringement.
- bayindirh 3y ago
  It's generally tucked under Fair Use doctrine because "It's for the science", until it doesn't (looking at commercial AI non-profits). Then "they're doing something amazing, they don't need permission, and the cat is already out of the bag, and similar musings". Seriously, it's both copyright infringement, and unethical. This is why I don't use any of the popular AI tools, or even AI add-ons in Evernote, Notion, etc. They all link back to the usual suspects.
  - layer8 3y ago
    I’m talking about distributing the corpus, which by itself is not bound to any particular usage.
    - bayindirh 3y ago
      It's again copyright infringement. If I share a copyrighted ebook by accident, any and every cloud provider will ban my account with no warning and recourse. Open science repositories would take down the "dataset" immediately (or at least limit its access) if a copyright holder brings the matter to the eyes of the admins.
  - CobrastanJorji 3y ago
    > "They're doing something amazing, they don't need permission, and the cat is already out of the bag" Ah, the Uber theory of law. Works surprisingly well for some reason.
    - bayindirh 3y ago
      Probably due to Murphy's Golden Law of Golden Laws: Who has the gold makes the laws.
  - Grimblewald 3y ago
    The question then becomes do these concerns remain even for AI that cannot reproduce origional works? Does what does that mean for us? When we read things, or interact with Amy information for that matter, it changes us and how we do things. If you consume art it will forever influence art you produce yourself. Are these copyright infringements also? I can see the problem where direct and faithful replication is possible but where it isn't is there still a problem? Or is the automatable aspect, the scale at which it can occur, that is the problem?
    - bayindirh 3y ago
      The difference is what you mix, and the amounts of things you mix. As a human you mix many more inputs, plus your emotions, plus the things you consume to create it. Moreover, what you can consume and how perfectly you can consume is limited by our innate limits. An AI system consumes something perfectly, then ingrains it into its weights perfectly, and becomes capable of imitating the same thing perfectly. Plus, ther are no other internal or external factors which affect these "generation" over time. Hence, it mixes and reproduces based on what it consumed, solely. I might get inspired by people, and add my own values to it, iterate over it and diverge from what I'm inspired from ultimately to create my own style. AI doesn't work like that. Also, if I do the same amount of inspiration with the same precision and accuracy, I'll be neck deep in accusations and lawsuits (for the right reasons). As a result, just because we fail to ask the right questions to reproduce the training data verbatim or almost verbatim doesn't mean that the information is not there. At the end, a neural network is a compression algorithm which encodes data in terms of weights. Given the correct input, you can regenerate the training data as is. Unless you have special abilities, you can't read 500 books an hour, remember them perfectly, and generate derivative works by mashing all of them together. If I do and try to sell a novel, I'll be ridiculed no end. If I write a Ph.D. the same way and try to defend it, I'll be banned from academia for three lifetimes at least. For more elaboration on the subject, see [0]. [0]: https://news.ycombinator.com/item?id=39188463 https://news.ycombinator.com/item?id=39188463
      - miki123211 3y ago
        The myth that AI models store all its training data in its weights verbatim is as widespread as it is false. In fact, if this were the case, deep neural networks would be considered far better compression algorithms than anything we have on the market right now, by literal orders of magnitude. If you divide Stable Diffusion's file size by the number of images used to train it, you get something like 1.2 bits per image, and it is physically impossible to get this kind of a compression ratio. The actual problem with AI is that it sometimes plagiarizes random fragments of the work it is trained on, even if it is not the user's intend, and we currently don't really know how to fully prevent this.

layer8 3y ago

I see, thanks. Yes, in that case, I don’t see how this can possibly not constitute copyright infringement.

bayindirh 3y ago
It's generally tucked under Fair Use doctrine because "It's for the science", until it doesn't (looking at commercial AI non-profits). Then "they're doing something amazing, they don't need permission, and the cat is already out of the bag, and similar musings". Seriously, it's both copyright infringement, and unethical. This is why I don't use any of the popular AI tools, or even AI add-ons in Evernote, Notion, etc. They all link back to the usual suspects.
- layer8 3y ago
  I’m talking about distributing the corpus, which by itself is not bound to any particular usage.
  - bayindirh 3y ago
    It's again copyright infringement. If I share a copyrighted ebook by accident, any and every cloud provider will ban my account with no warning and recourse. Open science repositories would take down the "dataset" immediately (or at least limit its access) if a copyright holder brings the matter to the eyes of the admins.
- CobrastanJorji 3y ago
  > "They're doing something amazing, they don't need permission, and the cat is already out of the bag" Ah, the Uber theory of law. Works surprisingly well for some reason.
  - bayindirh 3y ago
    Probably due to Murphy's Golden Law of Golden Laws: Who has the gold makes the laws.
- Grimblewald 3y ago
  The question then becomes do these concerns remain even for AI that cannot reproduce origional works? Does what does that mean for us? When we read things, or interact with Amy information for that matter, it changes us and how we do things. If you consume art it will forever influence art you produce yourself. Are these copyright infringements also? I can see the problem where direct and faithful replication is possible but where it isn't is there still a problem? Or is the automatable aspect, the scale at which it can occur, that is the problem?
  - bayindirh 3y ago
    The difference is what you mix, and the amounts of things you mix. As a human you mix many more inputs, plus your emotions, plus the things you consume to create it. Moreover, what you can consume and how perfectly you can consume is limited by our innate limits. An AI system consumes something perfectly, then ingrains it into its weights perfectly, and becomes capable of imitating the same thing perfectly. Plus, ther are no other internal or external factors which affect these "generation" over time. Hence, it mixes and reproduces based on what it consumed, solely. I might get inspired by people, and add my own values to it, iterate over it and diverge from what I'm inspired from ultimately to create my own style. AI doesn't work like that. Also, if I do the same amount of inspiration with the same precision and accuracy, I'll be neck deep in accusations and lawsuits (for the right reasons). As a result, just because we fail to ask the right questions to reproduce the training data verbatim or almost verbatim doesn't mean that the information is not there. At the end, a neural network is a compression algorithm which encodes data in terms of weights. Given the correct input, you can regenerate the training data as is. Unless you have special abilities, you can't read 500 books an hour, remember them perfectly, and generate derivative works by mashing all of them together. If I do and try to sell a novel, I'll be ridiculed no end. If I write a Ph.D. the same way and try to defend it, I'll be banned from academia for three lifetimes at least. For more elaboration on the subject, see [0]. [0]: https://news.ycombinator.com/item?id=39188463 https://news.ycombinator.com/item?id=39188463
    - miki123211 3y ago
      The myth that AI models store all its training data in its weights verbatim is as widespread as it is false. In fact, if this were the case, deep neural networks would be considered far better compression algorithms than anything we have on the market right now, by literal orders of magnitude. If you divide Stable Diffusion's file size by the number of images used to train it, you get something like 1.2 bits per image, and it is physically impossible to get this kind of a compression ratio. The actual problem with AI is that it sometimes plagiarizes random fragments of the work it is trained on, even if it is not the user's intend, and we currently don't really know how to fully prevent this.