5 ms·
wouldnt that mean more vram is required to load the model? they are claiming it will still work on 8 gb cards.
by darqwolff 3y ago
wouldnt that mean more vram is required to load the model? they are claiming it will still work on 8 gb cards.
- Filligree 3y agoStable Diffusion 1/2 were made to run on cards with as little as 3GB of memory. Using the same techniques, yes, this will fit in 8.
- brucethemoose2 3y agoI am guessing 8 bit quantization will be a thing for SDXL. It should be easy(TM) with bitsandbytes, or ML compiler frameworks.
- GaggiX 3y agobitsandbytes is only used during training with these models tho (the 8-bit Adamw) quantizing the weights and the activations to a range of 256 values when the model needs to output a range 256 values creates noticeable artifacts as they are not going to map 1-to-1.
- liuliu 3y agoDraw Things recently released a 8-bit quantized SD model that has comparable output as the FP16. It does use k-means based LUT and separate weights into blocks to minimize quantization errors.
- GaggiX 3y agoI was going to search on the internet about it, but then I realized you are the author (and there is nothing online I think). I imagine that the activations are left in FP16 and the weights are converted in FP16 during inference, right? Btw very cool
- liuliu 3y agoYes, computes are carried out in FP16 (so there is no compute efficiency gains, might be latency reductions due to memory-bandwidth saving). These savings are not realized yet because no custom kernels introduced yet.