10 ms·
The Illustrated Stable Diffusion
- jmartrican 4y agoSo its like the How to Draw an Owl meme.
- minimaxir 4y agoSD has made this meme into a reality, given how easy it is to take a sketch and use img2img to get something workable out of it.
- thunderbird120 4y agoYou can do that, yes https://0x0.st/oJVK.webm https://0x0.st/oJVK.webm
- marshray 4y agoNope, still don't understand it. :-/
- Waterluvian 4y agoCloser. But I still get lost when words like “tensor” are used. “structured lists of numbers” really doesn’t seem to explain it usefully. This reminds me that explaining seemingly complex things in simple terms is one of the most valuable and rarest skills in engineering. Most people just can’t. And often because they no-longer remember what’s not general knowledge. You end up with a recursive Feynmannian “now explain what that means” situation. This is probably why I admire a whole bunch of engineering YouTubers and other engineering “PR” people for their brilliance at making complex stuff seem very very simple.
- netruk44 4y agoIf it helps you to understand at all, assuming you have a CS background, any time you see the word "tensor" you can replace it with "array" and you'll be 95% of the way to understanding it. Or "matrix" if you have a mathematical background. Whereas CS arrays tend to be 1 dimensional, and sometimes 2 dimensional, tensors can be as many dimensions as you need. A 256x256 photo with RGB channels would be stored as a [256 x 256 x 3] tensor/array. If you want to store a bunch of them? Add a dimension to store each image. Want rows and columns of images? Make the dimensions [width x height x channels x rows x columns].
- minimaxir 4y agoA more practical example of the added dimensionality of tensors is the addition of a batch dimension, so a 8 image batch per training step would be a (8, 256, 256, 3) tensor. Tools such as PyTorch's DataLoader can efficiently collate multiple inputs into a batch.
- Waterluvian 4y agoThis helps. Thank you. Any advice on where to look to understand why the word tensor was used?
- deleted 4y ago[deleted]
- avereveard 4y agoTo reduce it a little, matrix holds numbers, tensor holds whatever. Numbers. Vectors. Operations.
- yreg 4y agoSo it is math lingo for array.
- squeaky-clean 4y agoIn math tensor has a more specific meaning than just an array, but in machine learning lingo, tensor just means an array or multidimensional array. That base is true in math as well, but it can also be a single value (rank zero tensor, aka a scalar. Sometimes ML will call this a tensor as well) and has to obey certain rules about linearity for some operations applied to it.
- sva_ 4y agoIt should be noted that it doesn't have all to much to do with the rigorous mathematical definition of a Tensor.
- jayalammar 4y ago
- 6gvONxR4sf7o 4y agoYou’re talking like using jargon makes something a bad explanation, but maybe you just aren’t the audience? Why not use words like that if it’s a super basic concept to your intended audience?
- Waterluvian 4y agoI saw the scientific term, “Text Understander” and wrongly thought I was the audience.
- deleted 4y ago[deleted]
- TigeriusKirk 4y agoOne of my favorite moments in Geoffrey Hinton's otherwise pretty info-dense Coursera neural network class was when he said- "To deal with a 14-dimensional space, visualize a 3-D space and say 'fourteen' to yourself very loudly. Everyone does it."
- TuringTest 4y agoKudos to this. It also helps to think of a three-dimensional space bounded by the sides of a box, and to think of another 11 boxes stacked on top of each other. Then, you can visualize orthogonal on dimensions higher than 3 by throwing wires between equivalent points in two boxes.
- dr_dshiv 4y agoOr imagine a spreadsheet with 14 columns, amiright?
- TuringTest 4y agoTechnically true, but you're leaving out any possible visual insights if you do that.
- dr_dshiv 4y agoI don’t know. I feel that, for most people, it is the most visual way to see in many dimensions. That is, it’s an insight—rather than having to imagine a 5-d cube or whatever, it’s just a spreadsheet, no big deal.
- TuringTest 4y agoYes, but they see it as a data dump, without perceiving any spacial connections between the vectors represented by each row. The point of spatial reasoning is using our minds inner eye to visualize those connections intuitively, and the tabular data dump has nothing of it.
- Waterluvian 4y agoI should have added that the images/figures really help. I think I’m about there.
- pdntspa 4y agoI got lost at the word 'tensor' too, but then I just googled it and skilled up... But simply put, a 'tensor' is a 3+-dimensional array of floats. Or a stack of matrices.
- paulgrant999 4y ago
- deleted 4y ago[deleted]
- beernet 4y agoI think this post explains tensors fairly well: https://www.kdnuggets.com/2018/05/wtf-tensor.html https://www.kdnuggets.com/2018/05/wtf-tensor.html Quote: "A tensor is a container which can house data in N dimensions. Often and erroneously used interchangeably with the matrix (which is specifically a 2-dimensional tensor), tensors are generalizations of matrices to N-dimensional space."
- minimaxir 4y agoHugging Face's diffusers library and explainer Colab notebook (https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/diffusers_intro.ipynb https://colab.research.google.com/github/huggingface/noteboo...) are good resources on how diffusion works in practice codewise.
- jayalammar 4y agoAgreed. "Stable Diffusion with Diffusers" and "The Annotated Diffusion Model" were excellent and are linked in the article. The code in Diffusers was also a good reference.
- torbTurret 4y agoLove the visual explainers for machine learning nowadays. The author has more here: https://jalammar.github.io/ https://jalammar.github.io/ Amazon has some highly interactive ones here: https://mlu-explain.github.io/ https://mlu-explain.github.io/ Google had: distill.pub Hope to see education in this space grow more.
- culi 4y ago> Google had: distill.pub Wait I thought they were just taking a break. Don't tell me it's being killed
- vanjajaja1 4y agoThis is the perfect level of description, thank you. Looking forward to checking out more of your work.
- jerpint 4y agoEverytime I need a refresher on transformers, I read the same author's post on transformers. Looking forward to this one!
- renewiltord 4y agoIsn't it really cool? It's like the AI is asking itself what shapes the clouds are making and whether the moon has a face, over and over again.
- mkaic 4y agoThis is one of the best analogies I've heard for denoising diffusion models. I'm totally stealing this the next time my parents ask me how it works :P
- coldcode 4y agoI find SD to be amazing technology, but it still (mostly) sucks at producing "intelligent" images. It basically fancy math that turns noise into images (from the opposite it trained on) but still has no idea what it is producing. If you run it long enough you eventually get lucky and find a gem. I like to try "George Washington riding a Unicorn in Times Square"; I've so far never gotten anything a first year art student can draw. I wonder how long it will take before something more "AI" than "ML" will have an understanding even close to what a simple human brain can process. In the meantime it's fun to play with it, plus I'd like to better understand the noise training process.
- minimaxir 4y agoWith SD, you have to use modifier quality/positional/artist keywords, as vanilla inputs give the model too much freedom.
- jw1224 4y ago> "George Washington riding a Unicorn in Times Square" The “secret” to Stable Diffusion (and other CLIP-based models) is being as descriptive as possible. This prompt, whilst easy for humans to imagine, actually has a whole lot of ambiguity baked in. How high is the unicorn flying? Is the unicorn even flying, or on the ground? How old is George Washington? What visual style is the image in? Is the image from the perspective of a pedestrian at ground level, or from up at skyscraper level? The more ambiguous the prompt, the less cohesive the image. To demonstrate, here’s 4 renders from your original prompt: https://imgur.com/a/Jo4qfOp https://imgur.com/a/Jo4qfOp And here’s 4 using the prompt “George Washington riding a unicorn in Times Square, cinematic composition, concept art, digital illustration, detailed”: https://imgur.com/a/lB36JqC https://imgur.com/a/lB36JqC Certainly not perfect, but for an additional 15 seconds of effort, far better.
- yieldcrv 4y agoWhat are you guys currently using for Stable Diffusion on OSX with M1? There are so many variants and forks that I don't know which one to install any more. Something that takes advantage of Metal and the CPU cores. Any that retains the "upload a sketch and then add a description" feature?
- pram 4y agoI'm using InvokeAI. Follow the instructions and it will work flawlessly. https://github.com/invoke-ai/InvokeAI https://github.com/invoke-ai/InvokeAI
- subdane 4y agohttps://github.com/divamgupta/diffusionbee-stable-diffusion-ui https://github.com/divamgupta/diffusionbee-stable-diffusion-...
- jaimex2 4y agoI just went ahead and got myself a PC with a good graphics card.
- swyx 4y agoi've been collecting other explanations of how SD works here: https://github.com/sw-yx/prompt-eng#sd-model-values https://github.com/sw-yx/prompt-eng#sd-model-values
- ilaksh 4y agoIt says the final output before pixel space is 64x64x4? How can that be enough information?
- l33tman 4y agoThe autoencoder that maps between that and the 512x512x3 RGB space was trained together with the model, so it is specialized in upscaling the 64x64x4 info to pixel space for this particular purpose. It's "just" a factor of 48 (de)compression..
- astrange 4y agoAnd remember we only care about 8 bits of the output (actually less due to JPEG compression) but each latent value is a 32-bit float.
- ilaksh 4y agoOh so each dimension is measured in 64 bit numbers? So the total bytes is (64*8) * (64*8) * (4 *8 )?*
- astrange 4y agoI think it’s considered an implementation detail so it’s just not mentioned. A lot of models still work if you round the values to FP16, which personally I find surprising.
- Imnimo 4y agoThe way I think of it, we have a 512x512x3 target, so that's 48x the information. I don't think it's unreasonable to say that far less than 1/48th of the space of 512x512x3 outputs are natural images (meaning an image that might actually exist, rather than meaningless pixels). So if we think about that 64x64x4 tensor as telling us what in the smaller space of natural images we should draw, it seems like plenty of information. Especially since we have the information stored weights of the output network also.
- minism 4y agoGreat overview, I think the part for me which is still very unintuitive is the denoising process. If the diffusion process is removing noise by predicting a final image and comparing it to the current one, why can't we just jump to the final predicted image? Or is the point that because its an iterative process, each noise step results in a different "final image" prediction?
- deleted 4y ago[deleted]
- cgearhart 4y agoI’m pretty sure it’s a stability issue. With small steps the noise is correlated between steps; if you tried it in one big jump then you would essentially just memorize the input data. The maximum noise would act as a “key” and the model would memorize the corresponding image as the “value”. But if we do it as a bunch of little steps then the nearby steps are correlated and in the training set you’ll find lots of groups of noise that are similar which allows the model to generalize instead of memorizing.
- nullc 4y agoYou can think of it like solving a differential equation numerically. The diffusion model encodes the relationships between values in sensible images (technically in the compressed representations of sensible images). You can try to jump directly to the solution but the result won't be very good compared to taking small steps.
- astrange 4y agoResearch is still ongoing here, but it seems like diffusion models despite being named after the noise addition/removal process don't actually work because of it. There's a paper (which I can't remember the name of) that shows the process still works with different information removal operators, including one with a circle wipe, and one where it blends the original picture with a cat photo. Also, this article describes CLIP being trained on text-image pairs, but Google's Imagen uses an off the shelf text model so that part doesn't seem to be needed either.
- ginger2016 4y agoThis is awesome ! Thank you for creating it. I have been wanting to read about Stable Diffusion. I added this to my reading list in Safari.
- uptown 4y ago“We then compare the resulting embeddings using cosine similarity. When we begin the training process, the similarity will be low, even if the text describes the image correctly.” How is this training performed? How is accuracy rated?
- mwambua 4y agoFrom the paper on the CLIP embedder it appears that they use a form of contrastive loss that maximizes the cosine similarity between related images and prompts, but also minimizes the same between unrelated prompts & images. See section 2.3 of the CLIP paper: https://arxiv.org/pdf/2103.00020.pdf https://arxiv.org/pdf/2103.00020.pdf Also, the writeup on OpenAI's blog: https://openai.com/blog/clip/ https://openai.com/blog/clip/
- kuu 4y agoCosine similarity is a fixed way of comparing two vectors, so we can think of it as making a difference: A-B = d If d is close to 0, we say that both embeddings are similar. If d is close to 1, we say that both embeddings are different. Imagine we have the following data: - Image A and its description A - Image B and its description B We would generate the following dataset: - Image A & Description A. Expected label: 0 - Image B & Description B. Expected label: 0 - Image A & Description B. Expected label: 1 - Image B & Description A. Expected label: 1 The mixture of Image Y with Description Z with Y!=Z is what we call "negative sampling" If the model predicts 1 but the expected value was 0 (or the other way around), it's a miss, and therefore the model is "penalized" and has to adjust the weights; if the prediction matches the expectation, it's a success, the model is not modified. I hope this clears it
- spywaregorilla 4y agoI'd be curious to see an example gallery of image generation of the same vector scaled to different magnitudes. That is, 100% cosine similarity, but still hitting different points of the embedding space. The outcome vectors aren't normalized right? So there could be a hefty amount of difference in this space? Maybe not on concept, but perhaps on image quality?
- bogomipz 4y agoWhat a great post! The author has one on GPT as well which I thought was also a great read.
- BrainVirus 4y agohttps://jalammar.github.io/images/stable-diffusion/article-Figure3-1-1536x762.png https://jalammar.github.io/images/stable-diffusion/article-F... Can you say with a straight face that this image (from the original paper) was intended to explain rather than obfuscate?
- backpropaganda 4y agoYes, I can read it perfectly fine. It relies on a lot of notation that AI researchers are familiar with.
- deleted 4y ago[deleted]
- herval 4y agoIf you're in AI research, that's a pretty standard diagram
- MacsHeadroom 4y agoIt's as clear as wiring diagram for me. But I read a lot of ML literature.
- ultrasounder 4y agoKudos to Author for writing this. The author's other guide, https://jalammar.github.io/gentle-visual-intro-to-data-analysis-python-pandas/ https://jalammar.github.io/gentle-visual-intro-to-data-analy... is THE most approachable introduction to Pandas for Excel users like me who are "comfortable" with Python(disclaimer: Self taught EE/Hardware guy). I will definitely spend some time this weekend to go over this tome. Thanks again.
- wwarek 4y agoAnother very good (although less deep) explanation was published yesterday by Computerphile: https://www.youtube.com/watch?v=1CIpzeNxIhU https://www.youtube.com/watch?v=1CIpzeNxIhU
- cl3misch 4y agoI also thought about this video while reading the HN post. The Computerphile video completely omits the latent space, right? But instead spends a lot of time on the iterative denoising. Even though I like Computerphile a lot, I don't think this was the best tradeoff.