9 ms·

The "memory usage" section of the README highlights the surprising fact that image generation models need much less memory than text-based language models. Chat

• DaiPlusPlus 4y ago
  I have zero qualifications to talk about this subject, but from what little I do know it's because Stable Diffusion's network has "only" ~900m parameters, whereas ChatGPT has 175 billion parameters.
  • flangola7 4y ago
    I wonder what a 100B image model will create.
    • cyanf 4y ago
      Images, most likely.
    • tomohelix 4y ago
      I dread that. You heard about Loab? That stuff is haunting. I am scared of what a 100B param model can create in the deep recesses of its system.
      • DaiPlusPlus 4y ago
        I thought Loab was just a creeopypasta?
        • JeremyBanks 4y ago
          No, Loab was an extreme value in parameter space which certain inputs could converge too.
  • p-e-w 4y ago
    Sure, but that's just another way of saying SD needs less space than ChatGPT, right? The question is why an image-generating model needs so much fewer parameters than a text-generating model in order to produce useful results, when our everyday experience teaches us that images need much more storage space than text to convey similar information.
    • zamnos 4y ago
      I think that's because the mind can make more sense of a mismash of visual data and see patterns and pictures. Comparatively if drop words here or you can still make sense of what I wrote, but it's much more difficult for the reader to parse what I'm saying if I accidentally a few words. If my drawing has a clearly recognizable human figure but it has 7 fingers, it's still recognizable that it was trying to draw a human. LLMs have been able to generate words for years now. Hell, Tay was back in 2016. Making sure ChatGPT is able to answer the way it does (ie filter out bad things) is part of what makes it hard, and thus bigger, to implement. But having a cohesive readable output is what's hard and takes up more space. Plus, unless I missed it, we don't actually know how many gigabytes the model file for ChatGPT is.
    • tomohelix 4y ago
      >images need much more storage space than text to convey similar information That is because an image is a collection of bit that attempt to represent reality as it is. It probably is easier to relates blue as a "color archetype" when you literally have a collection of bits that mean literal "blue" all the time. In languages "blue" doesn't always mean the color. Texts are abstractions/coded form of reality. Your mind itself contains the decryption codes to translate text into what it actually means. That decryption code for text apparently is much bigger and harder for a machine to crack than interpreting images. Also, images are bigger than a text file because of the way data is stored. It probaby has nothing to do with the amount of information stored inside. A book about quantum physics can be smaller than an image of a cat for example.
    • famouswaffles 4y ago
      The mistake you're making is thinking of weights as an avenue to store training data. This is wrong. I've hopefully explained it more in another comment but weights/neurons are essentially configuration settings.
• POiNTx 4y ago
  Not an expert at all, but maybe because the accuracy of a LLM needs to be higher than Stable Diffusion. Pictures can just 'look' good. But text very quickly can be 'off'. Put one word in the wrong place and the entire thing doesn't make sense anymore. And there's overall context you need to take care of, can't repeat and can't go off the rails too much. A picture at the end of the day is a 512x512 grid and there's many many combinations that would pass as a good result.
  • sebastiennight 4y ago
    This!!! I think the secret reason is that, on first glance, we are totally OK with six fingers in the picture. On closer examination we might balk at this but we won't notice if the "brush stroke" in a painting is slightly off (e.g. discontinuous in color by a few hues of red here or there). In language, we would definitely notice a missing, or a Japanese word in the 中 of an English text. So maybe the best and largest current image is just equivalent to GPT-1 in precision... but we just don't notice because our brains "smooth out" the imperfection.
• braingenious 4y ago
  When running diffusion models locally you often generate a lot of garbage. People with eight fingers per hand, cats with four tails, etc. When that happens you chalk that up to bad prompting or just the “magic” of image generation being too arcane to know. When it comes to text, people don’t find “cats are an animal with four tails” an amusing statement in the same way that they do a drawing of one. The standard of acceptability is way higher.
  • p-e-w 4y ago
    I'm not convinced by this explanation. Small language models (similar in size to the Stable Diffusion network) don't just produce incorrect statements like "cats are an animal with four tails", they produce incoherent sentences with no relation to the text they are supposed to extrapolate. It's not that they are wrong, they don't even make sense much of the time. That's not true for SD. Yes, many of its output images have flaws, but the overall image usually shows the desired subject, and roughly resembles something a human artist might paint.
    • braingenious 4y ago
      > they produce incoherent sentences with no relation to the text they are supposed to extrapolate I think we’re agreeing here? Language models of the size of SD produce literally useless output. SD can produce kind of fun output that people can make use of sometimes?
• SanderNL 4y ago
  I too have zero qualifications, but I think “text” is slightly more complicated - from an information theoretical perspective - than we give it credit for. A letter carries significantly more information than a pixel. One word can change the meaning of the rest of the text. (“joke:”) I don’t think images have the same property.
  • chrisco255 4y ago
    So you're saying a word is worth 1000 pictures? How did we get this so wrong?
    • uoaei 4y ago
      In a sense, language is a clumsy facsimile of the concepts we mean to express, in that we search for words to express the ideas in our minds rather than the other way around. By contrast, an image represents precisely the concept it depicts, by definition. We forgot this about language by about the time of the Enlightenment era, when the intellectuals of the time thought that forcing everything to inhabit the structures of language (i.e., "rationality") represented the highest moral good one could achieve.
    • SanderNL 4y ago
      Non-verbal experiences are easier to communicate non-verbally. (Breaking news!) But, I'd like to see someone try to communicate the Bhagavad Gita or the history of F=ma in visuals using the same amount of bits as text.
    • v01dlight 4y ago
      They're categorically different media, it's not just a matter of quantity. You could sum up 1000 pictures of bananas with the word "bananas", or you could spend 1000 different words describing nuances and context in just one of those pictures. Something is lost (and something is gained) either way.
  • NovaDudely 4y ago
    Possibly it is because with things like Stable diffusion we give it a lot of passes when things don't exactly right. Images just have to be close enough. Text however if it is only a single word out, the whole meaning and readability can change. It needs a significantly larger data set to ensure clearer readability. Just a shot from the hip response on this one.
• p1esk 4y ago
  Image corresponds to a sentence or sometimes just a single word. What ChatGPT does is closer to video generation - a much harder task. I expect video generation models to be much larger than LLMs.
• nopinsight 4y ago
  Natural language can represent an abstraction of very diverse stuff and phenomena in the world. It can represent events (with time dimension) and interactions between entities and events, abstract and physical, and meta-interactions as well, at multiple layers of abstractions. 2D still images, at least the sort that humans are familiar with, are more limited in terms of representation power and thus more amenable to compression into network weights. Perhaps models representing 4D phenomena (3D entities with time dimension, e.g. videos of real 3D models) would be more comparable in size to natural language models. Since LLMs can also represent abstract and unreal entities, while 4D representation can represent more details, it's hard to say which kind of models is richer.
  • espe 4y ago
    exactly. visual information is more compressable than natural language: much of it boils down to locality, whereas language forms are highly pareto distributed plus the conceptual system is a huge hypergraph, so it's rather the opposite of "local" organisation of information.
• nighthawk454 4y ago
  Completely different model sizes? Stable Diffusion is a ~1B parameter model. The 7B parameter LLAMA model would more comparable in RAM usage. It's not accurate to say "LLAMA needs 130 GB" because there's more than one LLAMA. The hard part in general is it often doesn't work very well to train small model sizes directly. You can train a very large one and distill it down, in some cases to only 1% of the original size while retaining 99%+ quality. So clearly the 1% size model exists. However training it directly usually doesn't work nearly as well. Best I can tell no one knows for sure why. Big tech cos are training the largest models as no one else has the hardware/power/money/etc to do so. SO they tend to release the most massive ones that only they can make. There's also a sect in the ML community that thinks "scale" is the answer to the universe... Then, either the tech cos or the community will make multiple other sizes of models and those are the ones that normies can use.
  • Helmut10001 4y ago
    > There's also a sect in the ML community that thinks "scale" is the answer to the universe... Offtopic: Reminds me of Laurent Nottale and his controversial "Scale Relativity and Fractal Space-Time: A New Approach to Unifying Relativity and Quantum Mechanics"
  • espadrine 4y ago
    Model sizes are adjusted to reach a given quality. Many papers justified empirically the use of more neurons allocated to language than to visual understanding, like Imagen[0] (figure 4.a). as it yields better results. Debatably, it is also true in the human brain. To me, it could highlight that there is more entropy in language than in images. After all, games have a similar property: some games, like noughts and crosses, or checkers, can have the same model architecture reach optimal play with smaller sizes than are necessary for chess and go[1]. It is certainly true that language has a lot of rules (grammatical, vocabular, syntactic) which are necessary to master it, but irrelevant to having a mental model of a scene; it just sequences a state of things into a stream of symbols. Maybe an additional entropy to learn in language which is absent in images includes motion and emotion. [0]: https://arxiv.org/pdf/2205.11487.pdf https://arxiv.org/pdf/2205.11487.pdf [1]: https://arxiv.org/pdf/2104.03113.pdf https://arxiv.org/pdf/2104.03113.pdf
• antli35 4y ago
  If you want to do a good job of generating text, you have to develop a model of how the world works. For example, if you describe an experiment from a paper to ChatGPT and ask it to generate the results section of the paper, then ChatGPT probably needs to understand the phenomenon that the experiment is about and be able to model it to some degree in order to generate plausible results. If you think about ChatGPT in this way, then it is not just a text generator, but a world simulator. The more accurately you can simulate the world, the better you can generate text. I think this is where the model's size and complexity comes from. ChatGPT needs to know as much as it can about basically everything. Putting it more generally, the difficulty of a computation isn't necessarily correlated to the filesize of the end product of that computation. Imagine simulating the entire world to try to predict what next week's lottery drawing numbers are going to be. Would require an unimaginable amount of data and computation, yet the output will be just a couple numbers.
• famouswaffles 4y ago
  Machine learning models don't store training data. The space a picture takes is irrelevant. For instance, stable diffusion would be the same size if it trained on 1 billion images than if it trained on 200 million or even 1 image(or 0 images). Weights/parameters are configuration settings not training data storage. When weights/neurons/parameters are updated after each training loop, you are essentially updating configuration settings that direct generations, not storing any particular training text or image. Weights are what take up the space. The bigger the parameter size/the number of weights, the bigger the size of the model. Image generators don't need the huge parameter numbers text generators need to be useful. What they need to learn simply isn't as complex.
  • idle_zealot 4y ago
    > What they (image generation models) need to learn simply isn't as complex. This is the surprising part. People seem to intuit that images are richer and more complex than words; a picture is worth a thousand words. But apparently this isn't true? Or perhaps our training methods for text models are way worse than those we use for image models.
    • hakuseki 4y ago
      This is just a guess, but I don't think there's such a deep lesson here; language models and image models have simply been developed by mostly-different groups of researchers who chose different tradeoffs. In an alternate history it may very well have gone the other way around.
      • famouswaffles 4y ago
        I would disagree. We have image generation with a variety of architectures. Diffusion models aside, it still takes a lot less parameters to model State of the art image generators with transformers (eg Parti). Simplifying a bit, mapping (which is essentially the main goal of image generators and especially transformer generators) is just less complex than prediction. It's like how bilingual llms can be much better translators than traditional map this sentence to this sentence translators. https://github.com/ogkalu2/Human-parity-on-machine-translations https://github.com/ogkalu2/Human-parity-on-machine-translati...
• valine 4y ago
  I like to think of stable diffusion as decompressing a text prompt into a 512x512 image. In some sense the 10 word text prompt and seed contains the same amount of information as a 262k pixel image. You could say to a transformer a picture is only worth a few words. By extension a if every image in the stable diffusion dataset compresses down to a sentence of text, then stable diffusion needs "only" 260M images * 10 words worth of information to train. Gpt3 on the other hand was trained on 45TB of text data.
• richardw 4y ago
  Here's a very unbaked thought, sparked by your question: a huge amount of organisms of varying sizes can parse and react to visual representation, but the amount of organisms that can handle any kind of language is much smaller.
• Al-Khwarizmi 4y ago
  Why is this so surprising? Almost all animals can process images, while only humans and arguably a few more can process language. Language is clearly much more complex from a processing point of view, even if it takes less memory when stored. (I know image generation models also use language, but to a much simpler extent, at least for now).
  • seydor 4y ago
    Most if our cortex is devoted to vision, however
    • ace2358 4y ago
      While I think it’s nonsense comparing what we do when we learn and what a computer Program does, I’ll speculate on this. Our eyes are doing a lot of signal processes before the “image” hits our brain. My understanding is audio has less signal processing required before the “sound” hits our brain.
  • hgsgm 4y ago
    Why do you think humans are the most complex minds? Humans minds can't even process infrared imagery or carbon dioxide levels in the air like insects can.
• dangrover 4y ago
  This was counter-intuitive to me too! I was recently playing around with some of the LLMs that can run on consumer hardware (via KoboldAI, RWKV, etc) and, boy, are they not as good as ChatGPT despite consuming all my Mac's resources. Meanwhile, can get Stable Diffusion images in under a minute!
• bravura 4y ago
  Visual knowledge can be decomposed into a limited set of primitives and operations. Whereas textual knowledge can accumulate arbitrarily and is bounded only by time. It’s the difference between learning a book of sports facts (language) and what sports facts look like (vision). Another example would be those passages in the Bible that list all the things owned by some person. There is a simple grammar to generate those passages. But learning them required more memory.
• uoaei 4y ago
  The model is doing more than spitting out text, it's spitting out language. Language has much different statistical properties than random text, and we would expect that we would need a significant amount of complexity to avoid generating complete trash from what is effectively an inferred probability distribution over long passages of language-bearing text.
• _0ffh 4y ago
  I'll take the bait and speculate that the manifold of long but reasonable token sequences is more complex than the manifold of reasonable pictures.
• adr1an 4y ago
  Size models, as people already pointed out. Also: keep in mind that reaolution of the images generated by txt2img models it is usually around 512*512. If you wanted a 50 megapixel photo, the VRAM requirement would grow by a lot. Granted, the number of parameters in the model would still be the same.
• stravant 4y ago
  It's about precision. Image models can be very off and still produce a satisfying result. Consider that I could literally vary all the pixels in an image randomly by 10% and you'd just see it as a bit low quality but otherwise perfectly cohesive image. Language models have no such luck, the problem they're trying to solve is way "sharper", it's very easy for their results to be strictly wrong if they're off even a little bit. So you need a much larger model to get a sufficient level of "sharpness" for text.
  • uh_uh 4y ago
    Maybe another way to think of it is that the error correction part of image generation models is offloaded to the human visual cortex which is a very old evolutionary construct and thus had time to become very resilient? In case of text generation, maybe the error tolerance of the human brain is less developed as human-level language is a newer evolutionary invention. It'd be interesting if the parameter/complexity requirements are actually similar once you examine the system as a whole, meaning machine _and_ human brain.
    • Codesleuth 4y ago
      > image generation models is offloaded to the human visual cortex which is a very old evolutionary construct and thus had time to become very resilient This is a very important point. A group of my colleagues (who are not tech people) are much more impressed with the image generation models than with the chat interface, even though the images are often whacky or just wrong. Yet the fact that it tried is impressive to them, with their minds managing to fill in the blanks. I wonder how this compares to how a toddler speaks vs. paints/draws, which is typically better in the former than the latter. I'm both cases, we fill in the blanks in our minds.
      • v01dlight 4y ago
        Toddler speaking gets impressive/surprising quite fast, whereas the drawing usually does not. The most surprising thing about most toddler drawings is listening to the kid describe it or tell you about making it.
• MacsHeadroom 4y ago
  LLaMA-13B 4bit needs only 18GB of VRAM for GPT-3 175B level text generation. But your point stands. LLaMA-65B 4bit needs 36 GB of VRAM, but far exceeds GPT-3's capabilities and even takes on PaLM 540B. See: https://github.com/oobabooga/text-generation-webui/wiki/LLaMA-model#4-bit-mode https://github.com/oobabooga/text-generation-webui/wiki/LLaM... for 4bit setup instructions See also: The case for 4-bit precision, which shows effectively no output quality reduction for these 4bit quantization methods (and considerable speedup) https://arxiv.org/abs/2212.09720 https://arxiv.org/abs/2212.09720
  • sterlind 4y ago
    absolutely amazing. I'm stunned how fast quantization was done. do you think there's anything left to trim? like weight pruning, or LoRA, or I dunno, some kind of Huffman coding scheme that lets you mix 4-bit, 2-bit and 1-bit quantizations?
    • MacsHeadroom 4y ago
      I can't edit my comment now, but it's 30B that needs 18GB of VRAM. LLaMA-13B, GPT-3 175B level, only needs 10GB of VRAM with the GPTQ 4bit quantization. >do you think there's anything left to trim? like weight pruning, or LoRA, or I dunno, some kind of Huffman coding scheme that lets you mix 4-bit, 2-bit and 1-bit quantizations? Absolutely. The GPTQ paper claims negligible output quality loss with 3-bit quantization. The GPTQ-for-LLaMA repo supports 3-bit quantization and inference. So this extra 25% savings is already possible. As of right now GPTQ-for-LLaMA is using a VRAM hungry attention method. Flash attention will reduce the requirements for 7B to 4GB and possibly fit 30B with a 2048 context window into 16GB, all before stacking 3-bit. Pruning is a possibility but I'm not aware of anyone working on it yet. LoRa has already been implemented. See https://github.com/zphang/minimal-llama#peft-fine-tuning-with-8-bit https://github.com/zphang/minimal-llama#peft-fine-tuning-wit...
  • Der_Einzige 4y ago
    Effective 4bit quantization seems like evidence of the devil's existence to me. This shouldn't be possible.
• lysecret 4y ago
  All decent replies but, the truth is that it is mostly a quirk of transformer architecture which scales quadratically in the length of the sequence because transformers look at all pairwise combinations of the input tokens. So you can get the memory to whatever level you want to just increase the sequence length (also known as context):
  • Der_Einzige 4y ago
    Flash attention gets this down to linear time complexity with no loss of quality.
    • sebzim4500 4y ago
      No it doesn't, it just improves the constant factor by a lot. Unless you are thinking of block sparse attention, which can chance complexity (assuming you scale sparsity with size) but decreases quality.
• zirgs 4y ago
  SD needs a lot more ram if you want to generate anything larger than 1024x1024.
  • WithinReason 4y ago
    It doesn't if you fix the network's code
• muyuu 4y ago
  if the intuition is based on images taking more space to encode than text, then it's a false intuition because the size of models is not correlated to the encoding size of the individual data points - it's correlated instead to the amount of relevant choices the model can make and the complexity of the dynamics in the model I don't think general assertions like "language is more complicated" are congruent or meaningful, it really depends on what the model is trying to achieve; it's the complexity of that which will require a larger or smaller model
• chrisweekly 4y ago
  "A picture's worth 1,000 words"
• random-walker 4y ago
  It is intriguing. I would have guessed human language with all its structure, would require way less parameters. If one were to look at the possibility of a 400x400 image and say 1000 words that describes it, the image would be from a 160,000^(16M) dimensional space. Whereas the 1000 words would require 1000^(40000) dimensional space. Space of all possible words seems smaller than all possible images. True, that visual image has a lot of redundancies, meaning I can change a lot of pixels and still the person will say both the images are the same. Whereas if you look at language, if you change even a few characters, humans might recognize the change. But human language is very heavily structured. It is constrained by grammar, constrained by semantics ('purple banana danced on top of the super-scalar processor' is nonsense.) etc. So once you apply these, the search space seems to get much much more constrained. Images are also constrained, for example if you take a random data point in the above space, it will look like noise to us. The visually interesting sub-space is much smaller. You can even constrain by sort of stochastic visual grammar (see David Mumford's work). The idea being humans have faces, faces have eyes etc. So if you see a face, you are more likely to see co-occurring parts as well. So both of them have a more constrained space that we are really interested in (one can define your own version of this). Our training of models is to differentiate/generate them within this space. And the question is, is one of these spaces definitely much smaller than the other. I would have presumed the constrained visual space is much larger than the constrained text space. Thus my only answer to the current contradiction being, we seem to be doing better with vision models than in language models. It could partly be that since we are more sensitive to errors in output of text, thus it is harder to find simpler models. Another way to look at this. Let's look at training data. A human child might see 65M images (assuming 1 image per sec given temporal redundancy, 10hrs awake) by age of 5. Would have heard 50-100M words (assuming 20-30k words/day) and spoken a few million words and so 'trained' for 20-40k hours. And the child can speak reasonably well by this time and detect common objects etc. Stable diffusion was trained on 170 Million images (1-2 order mag diff from child) or 3x10^13 bits of info and trained for 150K GPU hours giving a 1 Billion parameters. GPT3 was trained on 600x10^9 tokens of info and trained for 900k GPU hours giving a 170 Billion parameter model. So it seems like stable diffusion is getting a lot better compression. About 1/30k vs 1/4 compression. Caveat: Human learning process is much more complex and more effective (as of now at-least). We also learn actively by interacting with the world by changing the world etc. Think of the child gazing at the apple and looking at it from different angles or creating gibberish sentences very close to actual sentences and getting precise adult correction. We have a model of the world and we reason about it and provide 'consistency guarantees' between various questions about it, correctness etc (again all these only to a certain extent). Try asking questions like "I have a nail on the wall that is parallel to the floor, now I hang a painting on the wall. How is the painting placed with respect to the floor". Even a child would answer this.
• sandworm101 4y ago
  Entropy. Take a colored pixel. It is connected to a four other pixels each could be one of say 256 colors but are more likely likely to be a shade similar to each other. So you have roughly 1000 options. But then take a given English word. How many possible words might come after the word "the"? The set of possible connections between words in a paragraph are larger than the set of possible pixel colors in a given image (a realworld image, not white noise/static/random pixels.)
  • hgsgm 4y ago
    A pixel is equivalent to character in complexity.
• anigbrowl 4y ago
  Turns out one picture really is worth 1000 words.
• razodactyl 4y ago
  Images are limited in required dimensions. You tweak some pixels and you’re done. Language on the other hand… every word you say needs to be matched probabilistically with the next to make sense… imagine how many ways this reply alone could have been formulated :)