5 ms·

Gives me an error: 'list' object cannot be interpreted as an integer By the way: What is currently the most capable text to image model on Hugging Face? I

by TekMol 3y ago

Gives me an error: 'list' object cannot be interpreted as an integer By the way: What is currently the most capable text to image model on Hugging Face? I tried stable-diffusion-xl-base-1.0 but it seems far behind what the Bing image creator is creating these days. Is open source falling behind again?

petercooper 3y ago
Bing is using OpenAI's DALLE-3 which is somewhat ahead right now. However, SDXL (Stable Diffusion XL) can produce some pretty fantastic results, but it helps a lot to use a refining model (https://huggingface.co/stabilityai/stable-diffusion-xl-refiner-1.0 https://huggingface.co/stabilityai/stable-diffusion-xl-refin...) at the end of the generation. The automatic1111 webui has a feature to do this automatically built in, but other services have their own approaches. (BTW, regarding the error you saw - use this instead https://huggingface.co/spaces/segmind/Segmind-Stable-Diffusion https://huggingface.co/spaces/segmind/Segmind-Stable-Diffusi... )
- dragonwriter 3y ago
  > Bing is using OpenAI's DALLE-3 which is somewhat ahead right now. A lot of that is using an LLM for prompt expansion and composition guidance, both of which can be done with SD just by looping in an LLM for those purposes, though there isn’t support in any of the popular UIs for LLM composition assiatance (there are recent papers denonstrating it, though, so I wouldn't be surprised to see extensions for popular UIs within a month.) EDIT: corrected to clarify that only composition assistance isn’t yet supported in popular UIs, because prompt expansion is built in to Fooocus.
Sharlin 3y ago
Bing's DALL-E 3 is bleeding edge and a huge leap forward in prompt comprehension. Explained by the fact that it uses GPT-4 as the text encoder, so… similar performance probably not available on your local SD instance in the near future.
- TekMol 3y ago
  What is a "text encoder" and why does a text-2-image net need it?
  - Sharlin 3y ago
    A text encoder is the part that takes the prompt, tokenizes it, and then turns the tokens into an embedding [1] (an n-dimensional vector in the image (or latent image) space). The embedding is then used to "push" the noise diffusion process towards the desired part of the image space. The encoder is a net distinct from the diffusion net (and if the diffusion is done in a latent space like SD does, there's yet another net, so-called VAE, responsible for the image<->latent mapping). SD uses CLIP [2] as its text encoder. At a high level, the SD architecture looks like this: noisy latent -. +----------+ +------+ `-> | UNet | +-----+ prompt -> | CLIP | -> embedding -> | diffuser | -> denoised latent -> | VAE | -> output image +------+ +----------+ +-----+ [1] See, eg. this article currently on the HN front page: https://simonwillison.net/2023/Oct/23/embeddings/ https://simonwillison.net/2023/Oct/23/embeddings/ [2] https://openai.com/research/clip https://openai.com/research/clip
  - brucethemoose2 3y ago
    Its what "interprets" the prompt to give to the unet (which does the actual diffusion). SDXL's better text encoder is partially why it can understand sentences better than SD 1.5, and you can swap it out for, say, BLIP for combined image + text prompting.
  - famouswaffles 3y ago
    They've explained it but i'll point out that Dalle-3 doesn't use GPT-4 for the text encoder but the T5-XXL. The huge leap mostly comes from text descriptions that aren't trash.
- famouswaffles 3y ago
  DALL-E 3 uses T5-XXL for the text encoder not GPT-4. The huge leap comes mostly from using text descriptions that aren't trash. https://cdn.openai.com/papers/dall-e-3.pdf https://cdn.openai.com/papers/dall-e-3.pdf
michaelt 3y ago
> Is open source falling behind again? My impression is there's a large open development community that has come up with some really clever stuff - fine-tuning, LoRAs, ControlNets, regional prompting, and so on. You can teach the model completely new concepts and styles at home, for <$1 in electricity. But it's all been built on top of a base model trained by Stability AI at a cost of $600k (or at least, it would have cost that at AWS GPU prices). And some imperfections like mangled hands, prompt concept bleed, the inability to generate clock faces and mirrors etc seem to go deep into the base model, and beyond the point that fine-tuning can fix. And there aren't many people with $600k of spare cash lying around to train a new base model. The outputs of the open source models are great despite such weaknesses IMHO - but the costs involved mean the open source community is fighting with one hand tied behind its back. Of course, open models will always win in some areas - like if you want to generate a picture of Xi Jinping.
- orbital-decay 3y ago
  > But it's all been built on top of a base model trained by Stability AI at a cost of $600k (or at least, it would have cost that at AWS GPU prices). The LAION dataset they used was notoriously poor quality, and the training process wasn't optimal by any measure, though. The costs of training are rapidly falling due to various optimizations. Take a look at Pixart-alpha [0]. They claim SDXL-comparable performance for just $26k in training from scratch, with just 600M parameters in the unet and 25M pictures in the training set. Supposedly they achieved this due to the high quality training set tagged by a third-party model. The weights got leaked recently and the claim looks believable. [0] https://pixart-alpha.github.io/ https://pixart-alpha.github.io/
  - loudmax 3y ago
    This looks very promising and the sample images are impressive, especially for the small model size. Unfortunately, according to their Readme, "Inference requires at least 23GB of GPU memory." So not something you can run on consumer hardware in its current state.
    - 3y ago
brucethemoose2 3y ago
I can't get decent quality out of SDXL in base diffusers. Which is unfortunate, as HF diffusers is fast and very hackable. But try it in Fooocus, with its sea of augmentations (including Dall E-like prompt expansion): https://github.com/lllyasviel/Fooocus https://github.com/lllyasviel/Fooocus
- TekMol 3y ago
  How do I use that on Hugging Face?
  - brucethemoose2 3y ago
    You cannot, but you can host instances on Replicate, Vast.ai and others.
orbital-decay 3y ago
Stable Diffusion's strength (one of) is in the ability to be guided by sketches and references, which is much faster and more expressive/precise than anything achievable with a text prompt if you're good at it. If you want that, you want SD, it has vast amounts of related tooling around it. If text-to-image is enough for you, you need one of the commercial image generators with better prompt understanding.
schleck8 3y ago
Open Source has been behind proprietary options since the Midjourney release, SD was never better in terms of fidelity. That's how things work in general, FOSS trails commercial offers.

by TekMol 3y ago

petercooper 3y ago

Bing is using OpenAI's DALLE-3 which is somewhat ahead right now. However, SDXL (Stable Diffusion XL) can produce some pretty fantastic results, but it helps a lot to use a refining model (https://huggingface.co/stabilityai/stable-diffusion-xl-refiner-1.0 https://huggingface.co/stabilityai/stable-diffusion-xl-refin...) at the end of the generation. The automatic1111 webui has a feature to do this automatically built in, but other services have their own approaches. (BTW, regarding the error you saw - use this instead https://huggingface.co/spaces/segmind/Segmind-Stable-Diffusion https://huggingface.co/spaces/segmind/Segmind-Stable-Diffusi... )

dragonwriter 3y ago
> Bing is using OpenAI's DALLE-3 which is somewhat ahead right now. A lot of that is using an LLM for prompt expansion and composition guidance, both of which can be done with SD just by looping in an LLM for those purposes, though there isn’t support in any of the popular UIs for LLM composition assiatance (there are recent papers denonstrating it, though, so I wouldn't be surprised to see extensions for popular UIs within a month.) EDIT: corrected to clarify that only composition assistance isn’t yet supported in popular UIs, because prompt expansion is built in to Fooocus.

Sharlin 3y ago

Bing's DALL-E 3 is bleeding edge and a huge leap forward in prompt comprehension. Explained by the fact that it uses GPT-4 as the text encoder, so… similar performance probably not available on your local SD instance in the near future.

TekMol 3y ago
What is a "text encoder" and why does a text-2-image net need it?
- Sharlin 3y ago
  A text encoder is the part that takes the prompt, tokenizes it, and then turns the tokens into an embedding [1] (an n-dimensional vector in the image (or latent image) space). The embedding is then used to "push" the noise diffusion process towards the desired part of the image space. The encoder is a net distinct from the diffusion net (and if the diffusion is done in a latent space like SD does, there's yet another net, so-called VAE, responsible for the image<->latent mapping). SD uses CLIP [2] as its text encoder. At a high level, the SD architecture looks like this: noisy latent -. +----------+ +------+ `-> | UNet | +-----+ prompt -> | CLIP | -> embedding -> | diffuser | -> denoised latent -> | VAE | -> output image +------+ +----------+ +-----+ [1] See, eg. this article currently on the HN front page: https://simonwillison.net/2023/Oct/23/embeddings/ https://simonwillison.net/2023/Oct/23/embeddings/ [2] https://openai.com/research/clip https://openai.com/research/clip
- brucethemoose2 3y ago
  Its what "interprets" the prompt to give to the unet (which does the actual diffusion). SDXL's better text encoder is partially why it can understand sentences better than SD 1.5, and you can swap it out for, say, BLIP for combined image + text prompting.
- famouswaffles 3y ago
  They've explained it but i'll point out that Dalle-3 doesn't use GPT-4 for the text encoder but the T5-XXL. The huge leap mostly comes from text descriptions that aren't trash.
famouswaffles 3y ago
DALL-E 3 uses T5-XXL for the text encoder not GPT-4. The huge leap comes mostly from using text descriptions that aren't trash. https://cdn.openai.com/papers/dall-e-3.pdf https://cdn.openai.com/papers/dall-e-3.pdf

michaelt 3y ago

> Is open source falling behind again? My impression is there's a large open development community that has come up with some really clever stuff - fine-tuning, LoRAs, ControlNets, regional prompting, and so on. You can teach the model completely new concepts and styles at home, for <$1 in electricity. But it's all been built on top of a base model trained by Stability AI at a cost of $600k (or at least, it would have cost that at AWS GPU prices). And some imperfections like mangled hands, prompt concept bleed, the inability to generate clock faces and mirrors etc seem to go deep into the base model, and beyond the point that fine-tuning can fix. And there aren't many people with $600k of spare cash lying around to train a new base model. The outputs of the open source models are great despite such weaknesses IMHO - but the costs involved mean the open source community is fighting with one hand tied behind its back. Of course, open models will always win in some areas - like if you want to generate a picture of Xi Jinping.

orbital-decay 3y ago
> But it's all been built on top of a base model trained by Stability AI at a cost of $600k (or at least, it would have cost that at AWS GPU prices). The LAION dataset they used was notoriously poor quality, and the training process wasn't optimal by any measure, though. The costs of training are rapidly falling due to various optimizations. Take a look at Pixart-alpha [0]. They claim SDXL-comparable performance for just $26k in training from scratch, with just 600M parameters in the unet and 25M pictures in the training set. Supposedly they achieved this due to the high quality training set tagged by a third-party model. The weights got leaked recently and the claim looks believable. [0] https://pixart-alpha.github.io/ https://pixart-alpha.github.io/
- loudmax 3y ago
  This looks very promising and the sample images are impressive, especially for the small model size. Unfortunately, according to their Readme, "Inference requires at least 23GB of GPU memory." So not something you can run on consumer hardware in its current state.
  - 3y ago

brucethemoose2 3y ago

I can't get decent quality out of SDXL in base diffusers. Which is unfortunate, as HF diffusers is fast and very hackable. But try it in Fooocus, with its sea of augmentations (including Dall E-like prompt expansion): https://github.com/lllyasviel/Fooocus https://github.com/lllyasviel/Fooocus

TekMol 3y ago
How do I use that on Hugging Face?
- brucethemoose2 3y ago
  You cannot, but you can host instances on Replicate, Vast.ai and others.

orbital-decay 3y ago

Stable Diffusion's strength (one of) is in the ability to be guided by sketches and references, which is much faster and more expressive/precise than anything achievable with a text prompt if you're good at it. If you want that, you want SD, it has vast amounts of related tooling around it. If text-to-image is enough for you, you need one of the commercial image generators with better prompt understanding.

schleck8 3y ago

Open Source has been behind proprietary options since the Midjourney release, SD was never better in terms of fidelity. That's how things work in general, FOSS trails commercial offers.