26 ms·

completely different scenarios. A pirated movie is marketed/sold as a copy of something, which is not fair use. An LLM just remembers/get inspired by what it co

completely different scenarios. A pirated movie is marketed/sold as a copy of something, which is not fair use. An LLM just remembers/get inspired by what it consumes

xyzzy_plugh 1y ago
I don't believe that's correct. The existence of filters to block potentially copyrighted materials contained in LLM outputs proves that they don't just "get inspired." It seems like it is very much a matter of fidelity.
crystal_revenge 1y ago
> An LLM just remembers/get inspired by what it consumes As mentioned in another comment, LLMs (and most popular machine learning algorithms) can be viewed, correctly, as compression algorithms which leverage lossy encoding + interpolation to force a kind of generalization. Your argument is that a video wouldn't count as pirated if the compression used for the pirated copy was lossy (or at least sufficiently lossy). The closest real world example would be the cases where someone records a the filming of a movie on their phone then uploads it. Such a copy is lossy enough that you can't produce anything really like the original, but my most definitions is still considered copyright.
- Workaccount2 1y ago
  They are in no way compression algorithms. They can be modeled like that in the same way you can model humans as lossy compression algorithms. You would never use a human to backup your financial reports, but the human might be able to give a good overview. You would never use an LLM to backup your financial reports, but they might be able to give a good overview. AI training data is disposable. There is nothing that could be called a compression algorithm that disposes all of the data you put into it. AI uses training data as examples of what the next token in a token sequence is. The examples are disposable reference points, not the model itself. That's how you get image models that are 20GB in size despite training on 20PB of data. It's 20PB of examples used to form the shape of a 20GB model. You could show it 5GB of training data or 500EB of training data and it would still be 20GB - because it is not a compression algo, it's a 20GB shape formed by external data.
  - crystal_revenge 1y ago
    > They are in no way compression algorithms. I'm sorry, but this a fundamentally incorrect view of machine learning (including, but not limited to transformers). From an information theoretic perspective the two are essentially identical with the exception that standard compression algorithms do not have a proper "loss" function other than just trying to minimize reconstruction loss with the resulting compression size. Here's a link to the section on the Wikipedia for more information if you'd like [0]. MacKay's Information Theory, Inference and Learning Algorithms is the standard full text treatment of this topic [1]. Ted Chiang's article "ChatGPT is a Blurry JPEG of the web" is pretty good "pop sci" exploration of this topic if you don't want to get too into the mathematics [2]. 0. https://en.wikipedia.org/wiki/Data_compression#Machine_learning https://en.wikipedia.org/wiki/Data_compression#Machine_learn... 1. https://www.inference.org.uk/itprnn/book.pdf https://www.inference.org.uk/itprnn/book.pdf 2. https://www.newyorker.com/tech/annals-of-technology/chatgpt-is-a-blurry-jpeg-of-the-web https://www.newyorker.com/tech/annals-of-technology/chatgpt-...
    - Workaccount2 1y ago
      >They can be modeled like that in the same way you can model humans as lossy compression algorithms Humans are totally capable of data compression. This will just devolved into a semantics game of what a data compressor is. LLMs were not developed to be, do not function as, and are not use as data compression utilities. Please, come knocking when a service provider exists that will use LLM's to compactly store your company data.
      - crystal_revenge 1y ago
        > LLMs were not developed to be, do not function as, and are not use as data compression utilities. Again, from a information theoretic view point, this is exactly what they are doing, how they where developed and how they function. I don't know any serious researcher in ML that would find this claim even remotely controversial. It's really not just "a semantics game", its a part of a foundational understanding of the topic. If you want to understand LLMs from this perspective, a good place to start is with an auto-encoder which does try to learn a standard compression algorithm, the move on to more sophisticated embedding models (found in a lot of recommender systems) which try to learn an additional objective on top of minimizing reconstruction error. You'll then see that Transformers and all other major NN architectures fall out of these basic principles. > Please, come knocking when a service provider exists that will use LLM's to compactly store your company data. This is literally what every vectordb company does right now, as well as all "chat with your docs" type startups.

completely different scenarios. A pirated movie is marketed/sold as a copy of something, which is not fair use. An LLM just remembers/get inspired by what it consumes

xyzzy_plugh 1y ago

I don't believe that's correct. The existence of filters to block potentially copyrighted materials contained in LLM outputs proves that they don't just "get inspired." It seems like it is very much a matter of fidelity.

crystal_revenge 1y ago

> An LLM just remembers/get inspired by what it consumes As mentioned in another comment, LLMs (and most popular machine learning algorithms) can be viewed, correctly, as compression algorithms which leverage lossy encoding + interpolation to force a kind of generalization. Your argument is that a video wouldn't count as pirated if the compression used for the pirated copy was lossy (or at least sufficiently lossy). The closest real world example would be the cases where someone records a the filming of a movie on their phone then uploads it. Such a copy is lossy enough that you can't produce anything really like the original, but my most definitions is still considered copyright.

Workaccount2 1y ago
They are in no way compression algorithms. They can be modeled like that in the same way you can model humans as lossy compression algorithms. You would never use a human to backup your financial reports, but the human might be able to give a good overview. You would never use an LLM to backup your financial reports, but they might be able to give a good overview. AI training data is disposable. There is nothing that could be called a compression algorithm that disposes all of the data you put into it. AI uses training data as examples of what the next token in a token sequence is. The examples are disposable reference points, not the model itself. That's how you get image models that are 20GB in size despite training on 20PB of data. It's 20PB of examples used to form the shape of a 20GB model. You could show it 5GB of training data or 500EB of training data and it would still be 20GB - because it is not a compression algo, it's a 20GB shape formed by external data.
- crystal_revenge 1y ago
  > They are in no way compression algorithms. I'm sorry, but this a fundamentally incorrect view of machine learning (including, but not limited to transformers). From an information theoretic perspective the two are essentially identical with the exception that standard compression algorithms do not have a proper "loss" function other than just trying to minimize reconstruction loss with the resulting compression size. Here's a link to the section on the Wikipedia for more information if you'd like [0]. MacKay's Information Theory, Inference and Learning Algorithms is the standard full text treatment of this topic [1]. Ted Chiang's article "ChatGPT is a Blurry JPEG of the web" is pretty good "pop sci" exploration of this topic if you don't want to get too into the mathematics [2]. 0. https://en.wikipedia.org/wiki/Data_compression#Machine_learning https://en.wikipedia.org/wiki/Data_compression#Machine_learn... 1. https://www.inference.org.uk/itprnn/book.pdf https://www.inference.org.uk/itprnn/book.pdf 2. https://www.newyorker.com/tech/annals-of-technology/chatgpt-is-a-blurry-jpeg-of-the-web https://www.newyorker.com/tech/annals-of-technology/chatgpt-...
  - Workaccount2 1y ago
    >They can be modeled like that in the same way you can model humans as lossy compression algorithms Humans are totally capable of data compression. This will just devolved into a semantics game of what a data compressor is. LLMs were not developed to be, do not function as, and are not use as data compression utilities. Please, come knocking when a service provider exists that will use LLM's to compactly store your company data.
    - crystal_revenge 1y ago
      > LLMs were not developed to be, do not function as, and are not use as data compression utilities. Again, from a information theoretic view point, this is exactly what they are doing, how they where developed and how they function. I don't know any serious researcher in ML that would find this claim even remotely controversial. It's really not just "a semantics game", its a part of a foundational understanding of the topic. If you want to understand LLMs from this perspective, a good place to start is with an auto-encoder which does try to learn a standard compression algorithm, the move on to more sophisticated embedding models (found in a lot of recommender systems) which try to learn an additional objective on top of minimizing reconstruction error. You'll then see that Transformers and all other major NN architectures fall out of these basic principles. > Please, come knocking when a service provider exists that will use LLM's to compactly store your company data. This is literally what every vectordb company does right now, as well as all "chat with your docs" type startups.