3 ms·

> Is open source falling behind again? My impression is there's a large open development community that has come up with some really clever stuff - fine-tuning

> Is open source falling behind again? My impression is there's a large open development community that has come up with some really clever stuff - fine-tuning, LoRAs, ControlNets, regional prompting, and so on. You can teach the model completely new concepts and styles at home, for <$1 in electricity. But it's all been built on top of a base model trained by Stability AI at a cost of $600k (or at least, it would have cost that at AWS GPU prices). And some imperfections like mangled hands, prompt concept bleed, the inability to generate clock faces and mirrors etc seem to go deep into the base model, and beyond the point that fine-tuning can fix. And there aren't many people with $600k of spare cash lying around to train a new base model. The outputs of the open source models are great despite such weaknesses IMHO - but the costs involved mean the open source community is fighting with one hand tied behind its back. Of course, open models will always win in some areas - like if you want to generate a picture of Xi Jinping.

orbital-decay 3y ago
> But it's all been built on top of a base model trained by Stability AI at a cost of $600k (or at least, it would have cost that at AWS GPU prices). The LAION dataset they used was notoriously poor quality, and the training process wasn't optimal by any measure, though. The costs of training are rapidly falling due to various optimizations. Take a look at Pixart-alpha [0]. They claim SDXL-comparable performance for just $26k in training from scratch, with just 600M parameters in the unet and 25M pictures in the training set. Supposedly they achieved this due to the high quality training set tagged by a third-party model. The weights got leaked recently and the claim looks believable. [0] https://pixart-alpha.github.io/ https://pixart-alpha.github.io/
- loudmax 3y ago
  This looks very promising and the sample images are impressive, especially for the small model size. Unfortunately, according to their Readme, "Inference requires at least 23GB of GPU memory." So not something you can run on consumer hardware in its current state.
  - orbital-decay 3y ago
    That's because they slapped a 11B transformer on top of it for better prompt understanding; you can run it quantized to 8 bit and it will take about 14GB total, which has been tested by some people. This can be further reduced, or replaced for another encoder, the initial code is typically poorly optimized in these models (SD 1.4 required 12GB at first, then it was reduced to 4 and even 2GB). The meaningful part of the model that actually generates the picture is even smaller than the SD 1.x.

orbital-decay 3y ago

> But it's all been built on top of a base model trained by Stability AI at a cost of $600k (or at least, it would have cost that at AWS GPU prices). The LAION dataset they used was notoriously poor quality, and the training process wasn't optimal by any measure, though. The costs of training are rapidly falling due to various optimizations. Take a look at Pixart-alpha [0]. They claim SDXL-comparable performance for just $26k in training from scratch, with just 600M parameters in the unet and 25M pictures in the training set. Supposedly they achieved this due to the high quality training set tagged by a third-party model. The weights got leaked recently and the claim looks believable. [0] https://pixart-alpha.github.io/ https://pixart-alpha.github.io/

loudmax 3y ago
This looks very promising and the sample images are impressive, especially for the small model size. Unfortunately, according to their Readme, "Inference requires at least 23GB of GPU memory." So not something you can run on consumer hardware in its current state.
- orbital-decay 3y ago
  That's because they slapped a 11B transformer on top of it for better prompt understanding; you can run it quantized to 8 bit and it will take about 14GB total, which has been tested by some people. This can be further reduced, or replaced for another encoder, the initial code is typically poorly optimized in these models (SD 1.4 required 12GB at first, then it was reduced to 4 and even 2GB). The meaningful part of the model that actually generates the picture is even smaller than the SD 1.x.