4 ms·

So, and this is an ELI5 kind of question I suppose. There must be something going on like "processing a kazillion images" and I'm trying to wrap my head around

by jrm4 4y ago

So, and this is an ELI5 kind of question I suppose. There must be something going on like "processing a kazillion images" and I'm trying to wrap my head around how (or what part of) that work is "offloaded" to your home computer/graphics card? I just can't seem to make sense of how you can do it at home if you're not somehow in direct contact with "all the data?" e.g. must you be connected to the internet, or "stable-diffusions servers" for this to work?

codefined 4y ago
All those 'kazillion' images are processed into a single 'model'. Similar to how our brain cannot remember 100% of all our experiences, this model will not store precise copies of all images it is trained off of. However, it will understand concepts, such as what a unicorn looks like. For StableDiffusion, the current model is ~4GB, which is downloaded the first time you run the model. These 4GB encode all the information that the model requires to derive your images.
ducktective 4y ago
As someone with ~0 knowledge in this field, I think this has to do with a concept called "transfer learning" in which you once train with that kazillion of images, then use that same "coefficients" for further run of the NN.
- palunon 4y ago
  Nah, transfer learning is when you take a trained model, and train it a little more to better fit your (potentially very different) problem domain. Such as training a cat/dog/etc recognition model on MRI scans. The goal is usually to have the more fundamental parts of your model already working and you thus need way less domain specific data. Here, you're not training anything, you're running the models (both the CLIP language model and the unet) in feedforward. That's just deploying your model, not transfer learning.
sC3DX 4y ago
What you interact with as the user is the model and its weights. The model (presumably some kind of convolutional neural network) has many layers, every layer has some set of nodes, and every node has a weight, which is just some coefficient. The weights are 'learned' during the model training where the model takes in the data you mention and evaluates the output. This typically happens on a super beefy computer and can take a long time for a model like this. As images are evaluated the output gets better the weights get adjusted accordingly. Now we as the user just need the model and the weights!
dwohnitmok 4y ago
This is the main reason why attempts to say that these sorts of AI are just glorified lookup tables, or even that they are simply tools that mash together a kazillion images together are very misleading. A kazillion images are used in training, but training consists of using those images to tune on the order of ~5 GB of weights and that is the entire size of the final model. Those images are never stored anywhere else and are discarded immediately after being used to tune the model. Those 5 GB generate all the images we see.
juliendorra 4y ago
That’s the interesting part: all the images generated are derived from a less than 4gb model (the trained weights of the neural network). So in a way, hundreds of billions of possible images are all stored in the model (each a vector in multidimensional latent space) and turned into pixels on demand (drived by the language model that knows how to turn words into a vector in this space) As it’s deterministic (given the exact same request parameters, random seed included, you get the exact same image) it’s a form of compression (or at least encoding decoding) too: I could send you the parameters for 1 million images that you would be able to recreate on your side, just as a relatively small text file.
- amelius 4y ago
  > As it’s deterministic (given the exact same request parameters, random seed included, you get the exact same image) it’s a form of compression (or at least encoding decoding) too: I could send you the parameters for 1 million images that you would be able to recreate on your side, just as a relatively small text file. For any input image? Or do you mean an image generated by the model?
  - juliendorra 4y ago
    I meant images generated by the model. Now that I think of it I could just send you the sampled vectors and you could feed that to the vector to image part.
    - jimmySixDOF 4y ago
      My understanding is that images will not be bit identical due to GPU physics and decimal precision. Images from the same seed may be for all practical intents and purposes indistinguishable - but there are some flipped bits involved.
      - fragmede 4y ago
        That's not my understanding. The same seed value to the device's random number generator should results in the exact same outputs - there's a bug being chased down in the MPS (MacOS) backend where the fixed random seed doesn't output the same image on different computers.
l33tman 4y ago
SD has 860M weights for the main workhorse part. At 16-bit precision that is only 1.6 GB of data, which in some very real sense has condensed the world's total knowledge of art and photography and styles and objects. It's not a search engine, it's self-contained and the closest analogy is that it's a very very knowledgable and skilled artist.
- password4321 4y ago
  Is there a smaller version of the model available (<4gb) intended for use with 16 bit precision?
  - minimaxir 4y ago
    Diffusers shows how to use the fp16 variant. https://github.com/huggingface/diffusers https://github.com/huggingface/diffusers
bootsmann 4y ago
You can think of it more like this: If I do 100 experiments of dropping stones at variable heights and measuring the time it takes for the stone to land on the ground I have enough datapoints to make a linear estimation of gravity by using linear regression. So based on my data I create a model that the time it takes for a stone to fall is sqrt(2h/9.81). Now if you want to figure out how long it takes for your stones to fall, you don’t need to redo all the experiments and can instead rely on the parameters I give you (say 9.81 in this case) to calculate it yourself. With these models it works exactly the same way. Someone dropped millions of rocks and created a formula of unbelievable complexity and what they now did is they released that formula with all their calculated parameters into the world. What you do when you ultimately use Stable Diffusion is you just calculate the result of this formula and that is your image. You never have to process those images.
- MattRix 4y ago
  This is exactly it. It’s pretty remarkable that it was trained on over 100 terabytes of images and yet the model has been distilled down to only 4gb.
  - visarga 4y ago
    Then maybe we should remind about this 25,000:1 ratio when an artist complains about his copyrights being abused. The model doesn't have space to actually copy his works inside, it can only memorise the equivalent of a thumbnail from each input. A very small thumbnail, scaled down 150:1 per width and height (square root of 25000). That's like a grain of rice on the screen.
    - DrSiemer 4y ago
      That's not how it works though. Instead of applying arbitrary content detail reduction, the model is an attempt to distill the core of what makes a particular artist (or phrase, face, object etc) unique. When programming, it will often take a long time and a lot of code to get to a few final lines that do what you want. You cannot say the final result is a "thumbnail" of all previous efforts. Rather, it is the apotheosis of it. Some artists spend decades developing a style that looks like a kid could do it as well. Still, there is something unique in there, that a trained eye will recognize. Converting that particular style to a formula and making that freely available is at least somewhat morally ambiguous.
culi 4y ago
A kazillion images are used to create/optimize a neural network (basically). What you're working with is the result of that training. These are the "weights"
mirekrusin 4y ago
It’s all offline in 4gb file on your local computer. It’s like mini brain trained to do just one/few specific tasks. Just like your own brain doesn’t need Wi-Fi to connect to global memory storage of everything you experienced since birth, same way this 4gb file doesn’t need anything extra.

by jrm4 4y ago

codefined 4y ago

All those 'kazillion' images are processed into a single 'model'. Similar to how our brain cannot remember 100% of all our experiences, this model will not store precise copies of all images it is trained off of. However, it will understand concepts, such as what a unicorn looks like. For StableDiffusion, the current model is ~4GB, which is downloaded the first time you run the model. These 4GB encode all the information that the model requires to derive your images.

ducktective 4y ago

As someone with ~0 knowledge in this field, I think this has to do with a concept called "transfer learning" in which you once train with that kazillion of images, then use that same "coefficients" for further run of the NN.

palunon 4y ago
Nah, transfer learning is when you take a trained model, and train it a little more to better fit your (potentially very different) problem domain. Such as training a cat/dog/etc recognition model on MRI scans. The goal is usually to have the more fundamental parts of your model already working and you thus need way less domain specific data. Here, you're not training anything, you're running the models (both the CLIP language model and the unet) in feedforward. That's just deploying your model, not transfer learning.

sC3DX 4y ago

What you interact with as the user is the model and its weights. The model (presumably some kind of convolutional neural network) has many layers, every layer has some set of nodes, and every node has a weight, which is just some coefficient. The weights are 'learned' during the model training where the model takes in the data you mention and evaluates the output. This typically happens on a super beefy computer and can take a long time for a model like this. As images are evaluated the output gets better the weights get adjusted accordingly. Now we as the user just need the model and the weights!

dwohnitmok 4y ago

This is the main reason why attempts to say that these sorts of AI are just glorified lookup tables, or even that they are simply tools that mash together a kazillion images together are very misleading. A kazillion images are used in training, but training consists of using those images to tune on the order of ~5 GB of weights and that is the entire size of the final model. Those images are never stored anywhere else and are discarded immediately after being used to tune the model. Those 5 GB generate all the images we see.

juliendorra 4y ago

That’s the interesting part: all the images generated are derived from a less than 4gb model (the trained weights of the neural network). So in a way, hundreds of billions of possible images are all stored in the model (each a vector in multidimensional latent space) and turned into pixels on demand (drived by the language model that knows how to turn words into a vector in this space) As it’s deterministic (given the exact same request parameters, random seed included, you get the exact same image) it’s a form of compression (or at least encoding decoding) too: I could send you the parameters for 1 million images that you would be able to recreate on your side, just as a relatively small text file.

amelius 4y ago
> As it’s deterministic (given the exact same request parameters, random seed included, you get the exact same image) it’s a form of compression (or at least encoding decoding) too: I could send you the parameters for 1 million images that you would be able to recreate on your side, just as a relatively small text file. For any input image? Or do you mean an image generated by the model?
- juliendorra 4y ago
  I meant images generated by the model. Now that I think of it I could just send you the sampled vectors and you could feed that to the vector to image part.
  - jimmySixDOF 4y ago
    My understanding is that images will not be bit identical due to GPU physics and decimal precision. Images from the same seed may be for all practical intents and purposes indistinguishable - but there are some flipped bits involved.
    - fragmede 4y ago
      That's not my understanding. The same seed value to the device's random number generator should results in the exact same outputs - there's a bug being chased down in the MPS (MacOS) backend where the fixed random seed doesn't output the same image on different computers.

l33tman 4y ago

SD has 860M weights for the main workhorse part. At 16-bit precision that is only 1.6 GB of data, which in some very real sense has condensed the world's total knowledge of art and photography and styles and objects. It's not a search engine, it's self-contained and the closest analogy is that it's a very very knowledgable and skilled artist.

password4321 4y ago
Is there a smaller version of the model available (<4gb) intended for use with 16 bit precision?
- minimaxir 4y ago
  Diffusers shows how to use the fp16 variant. https://github.com/huggingface/diffusers https://github.com/huggingface/diffusers

bootsmann 4y ago

You can think of it more like this: If I do 100 experiments of dropping stones at variable heights and measuring the time it takes for the stone to land on the ground I have enough datapoints to make a linear estimation of gravity by using linear regression. So based on my data I create a model that the time it takes for a stone to fall is sqrt(2h/9.81). Now if you want to figure out how long it takes for your stones to fall, you don’t need to redo all the experiments and can instead rely on the parameters I give you (say 9.81 in this case) to calculate it yourself. With these models it works exactly the same way. Someone dropped millions of rocks and created a formula of unbelievable complexity and what they now did is they released that formula with all their calculated parameters into the world. What you do when you ultimately use Stable Diffusion is you just calculate the result of this formula and that is your image. You never have to process those images.

MattRix 4y ago
This is exactly it. It’s pretty remarkable that it was trained on over 100 terabytes of images and yet the model has been distilled down to only 4gb.
- visarga 4y ago
  Then maybe we should remind about this 25,000:1 ratio when an artist complains about his copyrights being abused. The model doesn't have space to actually copy his works inside, it can only memorise the equivalent of a thumbnail from each input. A very small thumbnail, scaled down 150:1 per width and height (square root of 25000). That's like a grain of rice on the screen.
  - DrSiemer 4y ago
    That's not how it works though. Instead of applying arbitrary content detail reduction, the model is an attempt to distill the core of what makes a particular artist (or phrase, face, object etc) unique. When programming, it will often take a long time and a lot of code to get to a few final lines that do what you want. You cannot say the final result is a "thumbnail" of all previous efforts. Rather, it is the apotheosis of it. Some artists spend decades developing a style that looks like a kid could do it as well. Still, there is something unique in there, that a trained eye will recognize. Converting that particular style to a formula and making that freely available is at least somewhat morally ambiguous.

culi 4y ago

A kazillion images are used to create/optimize a neural network (basically). What you're working with is the result of that training. These are the "weights"

mirekrusin 4y ago

It’s all offline in 4gb file on your local computer. It’s like mini brain trained to do just one/few specific tasks. Just like your own brain doesn’t need Wi-Fi to connect to global memory storage of everything you experienced since birth, same way this 4gb file doesn’t need anything extra.