5 ms·

I always wondered why the vision models don't seem to be following the whole "scale up as much as possible" mantra that has defined the language models of the p

I always wondered why the vision models don't seem to be following the whole "scale up as much as possible" mantra that has defined the language models of the past few years (to the same extent). Even 3.5 billion parameters is absolutely nothing compared to the likes of GPT-3, 3.5, 4, or even the larger open-source language models (e.g. LLaMA-65B). Is it just an engineering challenge that no one has stepped up for yet? Is it a matter of finding enough training data for the scaling up to make sense?

naillo 3y ago
They often reference this paper as the motivation for that https://arxiv.org/pdf/2203.15556.pdf https://arxiv.org/pdf/2203.15556.pdf I.e. training with 10x data and 10x longer can yield as good models as a gpt-3 model but with fewer weights (according to the paper) and the same principle applies in vision.
brucethemoose2 3y ago
Diffusion is relatively compute intensive compared to transformers llms, and (in current implementation) doesn't quantize as well. A 70B parameter model would be very slow and vram hungry, hence very expensive to run. Also, image generation is more reliant on tooling surrounding the models than pure text prompting. I dont think even a 300B model would get things quite right through text prompting alone.
- amilios 3y ago
  Hmm this is a good point, diffusion requires several (many?) inference passes as you refine the noise into an image, right? Makes sense that this is more expensive to scale up. Thanks for the explanation!
  - brucethemoose2 3y ago
    Technically the llms require a pass for each token, but the passes are cheaper and benefit more from batching.
airgapstopgap 3y ago
Diffusion is more parameter-efficient and you quickly saturate the target fidelity, especially with some refiner cascade. It's a solved problem. You do not need more than maybe 4B total. Images are far more redundant than text. In fact, most interesting papers since Imagen show that you get more mileage out of scaling the text encoder part, which is, of course, a Transformer. This is what drives accuracy, text rendering, compositionality, parsing edge cases. In SD 1.5 the text encoder part (CLIP ViT-L/14) takes a measly 123M parameters.[1] In Imagen, it was T5-XXL with 4.6B [2]. I am interested in someone trying to use a really strong encoder baseline – maybe from a UL2-20B – to push this tactic further. Seeing as you can throw out diffusion altogether and synthesize images with transformers [3], there is no reason to prioritize the diffusion part as such. 1. https://forums.fast.ai/t/stable-diffusion-parameter-budget-allocation/101515/1 https://forums.fast.ai/t/stable-diffusion-parameter-budget-a... 2. https://arxiv.org/abs/2205.11487 https://arxiv.org/abs/2205.11487 3. https://arxiv.org/abs/2301.00704 https://arxiv.org/abs/2301.00704
- Etherlord87 3y ago
  > Images are far more redundant than text. "A picture is worth a thousand words" - I wonder how (in)accurate this popular saying turned out to be? :D
  - elpocko 3y ago
    I'm gonna go ahead and say in 2023, one detailed picture (512x512) is worth about 30 words.
    - SketchySeaBeast 3y ago
      I guess that depends on the prompt.
      - k12sosse 3y ago
        Do negative prompt tokens count as words?
- 3y ago
vitorgrs 3y ago
Do we know the amount of parameters Dall-e have these days, Firefly or Midjourney, etc? If we are talking about Stable Diffusion, the reality is that... more parameters mean it will be hard to run locally. And let me tell you something, the community around Stable Diffusion only cares with NSFW... And want local for that... Stable Diffusion 2 was totally boycotted by the community because they... banned NSFW from there. They had now to allow it again on SDXL. Also, more parameters mean it will be more expensive to community finetunners to train as well.

naillo 3y ago

They often reference this paper as the motivation for that https://arxiv.org/pdf/2203.15556.pdf https://arxiv.org/pdf/2203.15556.pdf I.e. training with 10x data and 10x longer can yield as good models as a gpt-3 model but with fewer weights (according to the paper) and the same principle applies in vision.

brucethemoose2 3y ago

Diffusion is relatively compute intensive compared to transformers llms, and (in current implementation) doesn't quantize as well. A 70B parameter model would be very slow and vram hungry, hence very expensive to run. Also, image generation is more reliant on tooling surrounding the models than pure text prompting. I dont think even a 300B model would get things quite right through text prompting alone.

amilios 3y ago
Hmm this is a good point, diffusion requires several (many?) inference passes as you refine the noise into an image, right? Makes sense that this is more expensive to scale up. Thanks for the explanation!
- brucethemoose2 3y ago
  Technically the llms require a pass for each token, but the passes are cheaper and benefit more from batching.

airgapstopgap 3y ago

Diffusion is more parameter-efficient and you quickly saturate the target fidelity, especially with some refiner cascade. It's a solved problem. You do not need more than maybe 4B total. Images are far more redundant than text. In fact, most interesting papers since Imagen show that you get more mileage out of scaling the text encoder part, which is, of course, a Transformer. This is what drives accuracy, text rendering, compositionality, parsing edge cases. In SD 1.5 the text encoder part (CLIP ViT-L/14) takes a measly 123M parameters.[1] In Imagen, it was T5-XXL with 4.6B [2]. I am interested in someone trying to use a really strong encoder baseline – maybe from a UL2-20B – to push this tactic further. Seeing as you can throw out diffusion altogether and synthesize images with transformers [3], there is no reason to prioritize the diffusion part as such. 1. https://forums.fast.ai/t/stable-diffusion-parameter-budget-allocation/101515/1 https://forums.fast.ai/t/stable-diffusion-parameter-budget-a... 2. https://arxiv.org/abs/2205.11487 https://arxiv.org/abs/2205.11487 3. https://arxiv.org/abs/2301.00704 https://arxiv.org/abs/2301.00704

Etherlord87 3y ago
> Images are far more redundant than text. "A picture is worth a thousand words" - I wonder how (in)accurate this popular saying turned out to be? :D
- elpocko 3y ago
  I'm gonna go ahead and say in 2023, one detailed picture (512x512) is worth about 30 words.
  - SketchySeaBeast 3y ago
    I guess that depends on the prompt.
    - k12sosse 3y ago
      Do negative prompt tokens count as words?
3y ago

vitorgrs 3y ago

Do we know the amount of parameters Dall-e have these days, Firefly or Midjourney, etc? If we are talking about Stable Diffusion, the reality is that... more parameters mean it will be hard to run locally. And let me tell you something, the community around Stable Diffusion only cares with NSFW... And want local for that... Stable Diffusion 2 was totally boycotted by the community because they... banned NSFW from there. They had now to allow it again on SDXL. Also, more parameters mean it will be more expensive to community finetunners to train as well.