3 ms·
> LLMs are language models Aren't LLMs used to generate all these images like midjourney etc as well? Or is that a different type of model?
by hammock 3y ago
> LLMs are language models
Aren't LLMs used to generate all these images like midjourney etc as well? Or is that a different type of model?
- dartos 3y agoA image generator can and is built with the same underlying math, but different training data. LLM literally stands for “large language model”
- dragonwriter 3y agoLLMs can do a lot more than what a normal person would see as language by throwing different encoders and/or decoders than the standard text ones on either end of the model, though, and some models that do things that aren't superficially language work that way, so its a fair question to ask if you don’t specifically know how inage generation models work.
- dartos 3y agoI haven’t seen a model trained for language spit out an image given a pixel decoder. I have seen them make images by writing an svg. Do you have any links for something like a transformer model trained on language generating images with a different head?
- dragonwriter 3y agoNo, I've never seen that, either; I was just saying that given the number of other kinds of not-obviously-language things done that way, its a not-unreasonable question to have about inage generators.
- dartos 3y agoI wouldn’t call any transformer model an LLM. Just ones trained on languages. That’s why I said the same underlying math can be used for languages and images.
- hammock 3y agoI read somewhere today that a key benefit of transformers is they are effective at creating and using really long chains of dependencies. In my mind I interpreted that as “chatgpt feels like a conversation because the transformer model is emulating like 10, 20, 30 years of language practice /knowledge (not necessarily intelligence, but patterns and knowledge) with every query” meaning it goes way deeper than any neural net that came before it. Is that more or less accurate in my very layman’s understanding?
- dartos 3y agoThink of each token (roughly each word) as having a specific signal or tone. Each token after the first has a mix of the tones that came before. The transformer can guess what the tone of the next token is. The tones and how new tones are mixed from older tones are learned through training. Each tone is mathematically dependent on previous tones. It creates a dependency chain (more like a map, I think) in this way. “Feels like a conversation” is a rough metric to understand, but I think that feel mostly comes from how chatgpt is presented from a UX perspective. Personally I don’t like using words like “practice” or phrases like “20 years of knowledge” because they’re fuzzy and don’t really reflect what’s going on under the hood. Imo they make things harder to understand
- hammock 3y agoYeah conversation was my word for “better” but I didn’t mean it any deeper than that. “20 years of knowledge “ I just mean the map is bigger and more filled in I think we are saying along the same lines
- dartos 3y agoI’d recommend reading the paper “Attention is all you need” as it lays a lot of the foundational knowledge about transformers. I’m no math guru, so I had to read the paper like 5 or 6 times to wrap my head around it. I had to stop trying to understand how the math worked exactly and just accepted that it did, then it started to make sense. Now going back I can actually understand some of why the math works.
- hammock 3y agoYeah that is my confusion.. these image generators take language input but issue image output - is the imagery considered "language" in the context of an LLM? Are image generators transformers but not LLMs? Something else?
- dragonwriter 3y agoMostly, the current well-known image generation models are diffusion models that use a U-Net; newer ones (Pixart-α, Sora, Stable Diffusion 3.0) are starting to be diffusion transformers (DiT), where the U-Net is replaced with a transformer model.
- hammock 3y agoThank you. Why is it so hard for me to find this information by searching?
- dartos 3y agoYou must know which questions to ask and how to sift through all the hype marketing. A lot of this info is in arxiv papers too. Not the easiest stuff to search for.
- dragonwriter 3y agoHype has a way of drowning out technical information from easy searchability without knowing some of the right terminology that, often, is what you would be trying to figure out if you were searching.
- dragonwriter 3y ago> Aren't LLMs used to generate all these images like midjourney etc as well? No (though the text encoder of text-to-image model is like part of some LLMs, and some UIs use a full LLM as a prompt preprocessor.)