4 ms·

You can think of it more like this: If I do 100 experiments of dropping stones at variable heights and measuring the time it takes for the stone to land on the

You can think of it more like this: If I do 100 experiments of dropping stones at variable heights and measuring the time it takes for the stone to land on the ground I have enough datapoints to make a linear estimation of gravity by using linear regression. So based on my data I create a model that the time it takes for a stone to fall is sqrt(2h/9.81). Now if you want to figure out how long it takes for your stones to fall, you don’t need to redo all the experiments and can instead rely on the parameters I give you (say 9.81 in this case) to calculate it yourself. With these models it works exactly the same way. Someone dropped millions of rocks and created a formula of unbelievable complexity and what they now did is they released that formula with all their calculated parameters into the world. What you do when you ultimately use Stable Diffusion is you just calculate the result of this formula and that is your image. You never have to process those images.

MattRix 4y ago
This is exactly it. It’s pretty remarkable that it was trained on over 100 terabytes of images and yet the model has been distilled down to only 4gb.
- visarga 4y ago
  Then maybe we should remind about this 25,000:1 ratio when an artist complains about his copyrights being abused. The model doesn't have space to actually copy his works inside, it can only memorise the equivalent of a thumbnail from each input. A very small thumbnail, scaled down 150:1 per width and height (square root of 25000). That's like a grain of rice on the screen.
  - DrSiemer 4y ago
    That's not how it works though. Instead of applying arbitrary content detail reduction, the model is an attempt to distill the core of what makes a particular artist (or phrase, face, object etc) unique. When programming, it will often take a long time and a lot of code to get to a few final lines that do what you want. You cannot say the final result is a "thumbnail" of all previous efforts. Rather, it is the apotheosis of it. Some artists spend decades developing a style that looks like a kid could do it as well. Still, there is something unique in there, that a trained eye will recognize. Converting that particular style to a formula and making that freely available is at least somewhat morally ambiguous.
    - holoduke 4y ago
      It's the same as someone trying to mimic a style. Nothing wrong with that. Certainly not something you could get copyrights from.
      - DrSiemer 4y ago
        It is not the same as trying to mimic a style. It is cloning the essence of a style and making it readily available to anyone who asks for it. Sure, it's not copyright infringement, but you could argue that this takes away from the hardship the original artist had to go through to perfect their style.
        namarie 4y ago
        > Sure, it's not copyright infringement Which was precisely what the discussion was about. > But you could argue that this takes away from the hardship the original artist had to go through to perfect their style You could argue the same things about photoshop, a lot of other digital tools, drum machines, the photograph and the phonograph.
  - jrm4 4y ago
    Ah, I can step in here. Fair use might work but maybe not? If I were to argue against it, I'd probably compare something like a recording of music vs. a MIDI file. Same raw data scaling.
- rassibassi 4y ago
  Yes, and another reason for the small model size and the novelty of the underlying paper [1], is that the diffusion model is not acting on the pixel space but rather on a latent space. This means that this 'latent diffusion model' does not only learn the task at hand (image synthesis) but in parallel also a powerful lossy compression model via an outer auto encoder structure. Now, the number of weights (model size) can be reduced drastically as the inner neural network layers act on a lower dimensional latent space rather than a high dimensional pixel space. It's fascinating because it shows that deep learning at its core comes down to compression/decompression (encoding/decoding), with close relation to Shannon's Information Theory (e.g. source coding/channel coding/data processing inequality). [1] https://arxiv.org/abs/2112.10752 https://arxiv.org/abs/2112.10752
  - rassibassi 4y ago
    Correction: the auto encoder is pre-trained :)
  - jrm4 4y ago
    Oh, wow. Now that you mention how it's similar to lossy (if not the same as) compression it all makes a LOT of sense. This is great. I teach IT and I already do a bit on how lossy compression works, (e.g. hey, if you see a blue pixel and then another slightly darker one next to it, what's the NEXT likely to be?) and this is something of an extension of that.

MattRix 4y ago

This is exactly it. It’s pretty remarkable that it was trained on over 100 terabytes of images and yet the model has been distilled down to only 4gb.

visarga 4y ago
Then maybe we should remind about this 25,000:1 ratio when an artist complains about his copyrights being abused. The model doesn't have space to actually copy his works inside, it can only memorise the equivalent of a thumbnail from each input. A very small thumbnail, scaled down 150:1 per width and height (square root of 25000). That's like a grain of rice on the screen.
- DrSiemer 4y ago
  That's not how it works though. Instead of applying arbitrary content detail reduction, the model is an attempt to distill the core of what makes a particular artist (or phrase, face, object etc) unique. When programming, it will often take a long time and a lot of code to get to a few final lines that do what you want. You cannot say the final result is a "thumbnail" of all previous efforts. Rather, it is the apotheosis of it. Some artists spend decades developing a style that looks like a kid could do it as well. Still, there is something unique in there, that a trained eye will recognize. Converting that particular style to a formula and making that freely available is at least somewhat morally ambiguous.
  - holoduke 4y ago
    It's the same as someone trying to mimic a style. Nothing wrong with that. Certainly not something you could get copyrights from.
    - DrSiemer 4y ago
      It is not the same as trying to mimic a style. It is cloning the essence of a style and making it readily available to anyone who asks for it. Sure, it's not copyright infringement, but you could argue that this takes away from the hardship the original artist had to go through to perfect their style.
      - namarie 4y ago
        > Sure, it's not copyright infringement Which was precisely what the discussion was about. > But you could argue that this takes away from the hardship the original artist had to go through to perfect their style You could argue the same things about photoshop, a lot of other digital tools, drum machines, the photograph and the phonograph.
- jrm4 4y ago
  Ah, I can step in here. Fair use might work but maybe not? If I were to argue against it, I'd probably compare something like a recording of music vs. a MIDI file. Same raw data scaling.
rassibassi 4y ago
Yes, and another reason for the small model size and the novelty of the underlying paper [1], is that the diffusion model is not acting on the pixel space but rather on a latent space. This means that this 'latent diffusion model' does not only learn the task at hand (image synthesis) but in parallel also a powerful lossy compression model via an outer auto encoder structure. Now, the number of weights (model size) can be reduced drastically as the inner neural network layers act on a lower dimensional latent space rather than a high dimensional pixel space. It's fascinating because it shows that deep learning at its core comes down to compression/decompression (encoding/decoding), with close relation to Shannon's Information Theory (e.g. source coding/channel coding/data processing inequality). [1] https://arxiv.org/abs/2112.10752 https://arxiv.org/abs/2112.10752
- rassibassi 4y ago
  Correction: the auto encoder is pre-trained :)
- jrm4 4y ago
  Oh, wow. Now that you mention how it's similar to lossy (if not the same as) compression it all makes a LOT of sense. This is great. I teach IT and I already do a bit on how lossy compression works, (e.g. hey, if you see a blue pixel and then another slightly darker one next to it, what's the NEXT likely to be?) and this is something of an extension of that.