6 ms·
SVDQuant: 4-Bit Quantization Powers 12B Flux on a 16GB 4090 GPU with 3x Speedup
- deleted 2y ago[deleted]
- mesmertech 2y agoDemo on actual 4090 with flux schnell for next few hours: https://5jkdpo3rnipsem-3000.proxy.runpod.net/ https://5jkdpo3rnipsem-3000.proxy.runpod.net/ Its basically H100 speeds with 4090, 4.80it/s. 1.1 sec for flux schenll(4 steps) and 5.5 seconds for flux dev(25 steps). Compared to normal speeds(comfyui fp8 with "--fast" optimization") which is 3 seconds for schnell and 11.5 seconds for dev
- yakorevivan 2y agoHey, can you share the inference code please? Thanks..
- superkuh 2y agohttps://github.com/mit-han-lab/nunchaku https://github.com/mit-han-lab/nunchaku
- oneshtein 2y agoCannot compile it locally on Fedora 40: nunchaku/third_party/spdlog/include/spdlog/common.h(144): error: namespace "std" has no member "function" using err_handler = std::function<void(const std::string &err_msg)>; ^
- mesmertech 2y agoYea its a pain, I'm trying to make an api endpoint for a website I own, and working on a docker image. This is what I have for now that "just" works: the conda always yes thing makes sure that you can just paste the script and it all works instead of having to press "y" for each install. Also if you don't feel like installing a wheel from random person on the internet, replace that step with "pip install -e ." as the repo suggests. I compiled that one with cuda 12.4 cause that was the part takes the most time and is what most often seems to be breaking. Also I'm not sure if this will work on Fedora, I tried this on a runpod machine with 4090(apparently it only works on few cards, 3090, 4090, a100 etc) with Cuda 12.4 on host machine and "runpod/pytorch:2.4.0-py3.11-cuda12.4.1-devel-ubuntu22.04" this image as base. EDIT: using pastebin instead as HN doesn't seem to jive with code blocks: https://pastebin.com/zK1z0UdM https://pastebin.com/zK1z0UdM
- oneshtein 2y agoAlmost working: [2024-11-09 19:33:55.214] [info] Initializing QuantizedFluxModel [2024-11-09 19:33:55.359] [info] Loading weights from ~/.cache/huggingface/hub/models--mit-han-lab--svdquant-models/snapshots/d2a46e82a378ec70e3329a2219ac4331a444a999/svdq-int4-flux.1-schnell.safetensors [2024-11-09 19:34:01.432] [warning] Unable to pin memory: invalid argument [2024-11-09 19:34:02.143] [info] Done. terminate called after throwing an instance of 'CUDAError' what(): CUDA error: pointer does not correspond to a registered memory region (at /nunchaku/src/Serialization.cpp:32)
- mesmertech 2y agoprolly make sure your host machine cuda is also 12.4 and if not, update the other cuda versions I have on the pastebin to the one you have. I don't think it works with cuda 11.8 tho, remember trying it once but yea, can't help you outside of runpod, I haven't even tried this on my home PCs yet. for my usecase of serverless API, it seems to work
- bufferoverflow 2y agoDamn, it runs very fast.
- AzN1337c0d3r 2y agoIt's worth noting this is laptop 4090 GPU which is more like in the range of desktop 4070 performance.
- mesmertech 2y agoThis specific link I shared is the quant running on a 4090 I rented on runpod, I have no affiliation with the repo itself
- qeternity 2y agoThe compute differential between an H100 and a 4090 is not huge. The main single GPU benefits are larger memory (and thus memory bandwidth) and native fp8. But these matter less for diffusion models.
- mesmertech 2y agoThats what I thought as well, but FP8 is much faster on h100, like 2x-3x. You can check it/s here: https://github.com/aredden/flux-fp8-api https://github.com/aredden/flux-fp8-api Its why fal, replicate, pretty much all big diffusion api providers use h100 tldr; 4090 is max 3.51 it/s even with all the current optimizations. h100 is 11.5it/s with all optimizations, and even without its 6.1 it/s
- boroboro4 2y agoProviders use h100 because using 4090 in DCs is grey area, since Nvidia doesn't permit it. Paper discussing here is using 4 bit compute, which is 4x on 4090 in comparison with bf16 compute, while h100 doesn't have this at all (i.e. best you can get is 2x compute with fp8). So this paper will even out difference between those two to some extent. If to judge by theoretical numbers - H100 has 1979 TFLOPs fp8 compute, and 4090 has 1321 TOPS. Which puts it around ~65% of performance. Given the price of it ~$2K compared to H100s ~$30K this seems like a very good deal. But again, no 4090 in DCs.
- notarealllama 2y agoI'm convinced the path to ubiquity (such as embedded in smartphones) is quantization. I had to int4 a llama model to get it to properly run on my 3060. I'm curious, how much resolution / significant digits do we actually need for most genAI work? If you can draw a circle with 3.14, maybe it's good enough for fast and ubiquitous usage.
- sigmoid10 2y agoEarlier this year there was a paper from Microsoft where they trained a 1.58 bit (every parameter being ternary) LLM that matched the performance of 16 bit models. There's also other research that you can prune up to 50% of layers with minimal loss of performance. Our current training methods are just incredibly crude and we will probably look back on those in the future and wonder how this ever worked at all.
- llm_trw 2y agoNone of those papers actually use quantized training, they are all about quantized inference. Which is rather unfortunate as it means that the difference between what you can train locally and what you can run locally is growing ever larger.
- danielEM 2y agoIndeed. I think "AI gold rush" sucks anyone with any skills in this area into it with relatively good pay, so there are no, or almost no people outside of big tech and startups to counterbalance direction where it moves. And as a side note, big tech is and always was putting their agenda first in developing any tech or standards and that usually makes milking on investments as long as possible, not necessary moving things forward.
- llm_trw 2y agoThere's more to it than that. If you could train models faster, you’d be able to build larger, more powerful models that outperform the competition. The fact that Llama 3 is significantly over trained than what was considered ideal even three years ago shows there's a strong appetite for efficient training. The lack of progress isn’t due to a lack of effort. No one has managed to do this yet because no one has figured out how. I built 1-trit quantized models as a side project nearly a decade ago. Back then, no one cared because models weren’t yet using all available memory, and on devices where memory was fully utilized, compute power was the limiting factor. I spend much longer trying to figure out how to get 1-trit training to work and I never could. Of all the papers and people in the field I've talked to, no one else has either.
- xrd 2y agoCan someone explain this sentence from the article: Diffusion models, however, are computationally bound, even for single batches, so quantizing weights alone yields limited gains.
- flutetornado 2y agoGPU workloads are either compute bound (floating point operations) or memory bound (bytes being transferred across memory hierarchy.) Quantizing in general helps with the memory bottleneck but does not help in reducing computational costs, so it’s not as useful for improving performance of diffusion models, that’s what it’s saying.
- pkAbstract 2y agoExactly. The smaller bit widths from quantization might marginally decrease the compute required for each operation, but they do not reduce the overall volume of operations. So, the effect of quantization is generally more impactful on memory use than compute.
- superkuh 2y agoExcept in this case they quantized both the parameters and the activations leading to decreased compute time too.
- llm_trw 2y agoDiffusion requires a lot more computation to get results compared to transformers. Naively when I'm running a transformer locally I get about 30% GPU utilization, when I'm running a diffusion model I'm getting 100%. This means that the only saving you're getting in speed for a diffusion model is being able to do more effective flops since the floats are smaller, e.g. instead of doing one 32bit multiplication, you're doing 8 4bit ones. By comparison for transformers you not only gain the flop increase, but also the improvement in memory shuffling that they do, e.g. it also takes you 8 times less time to load the memory into working memory from vram. The above is a vast over simplification and in practice will have more asterisks than you can shake a stick at.
- DeathArrow 2y agoBut doesn't quantization give worse results? Don't you trade quality for memory footprint?
- timnetworks 2y agoThey're saying this method essential does not, even when mixed with low rank models on top. "Notably, while the original BF16 model requires per-layer CPU offloading on the 16GB laptop 4090, our INT4 model fits entirely in GPU memory, resulting in a 10.1× speedup by avoiding offloading." This is the whole magic, the rest of the workflow doesn't need to unload and flush memory, causing big delays for jobs.
- scottmas 2y agoPossible to run this in ComfyUI?
- vergessenmir 2y agoThe repo has sample code and it is fairly easy to create a node that will do it. You won't however have access to usual sampler, latent image, Lora nodes to do anything beyond basic t2i
- doctorpangloss 2y agoWhy? There is nothing to customize with Flux.
- thot_experiment 2y agoWhat do you mean there's nothing to customize with Flux, can you expand on this claim?
- djoldman 2y agoThis is one in a long line of posts saying "we took a model and made it smaller" and now it can run with different requirements. It is important to keep in mind that modifying a model changes the performance of the resulting model, where performance is "correctness" or "quality" of output. Just because the base model is very performant does not mean the smaller model is. This means that another model that is the same size as the new quantized model may outperform the quantized model. Suppose there are equal sized big models A and B with their smaller quantized variants a and b. A being a more performant model than B does not guarantee a being more performant than b.
- superkuh 2y agoNot really. They quantized the activations here with their inference program which decreased compute as well as RAM usage (and required bandwidth). That's a big step.
- ttul 2y agoWhile I think I agree that there are many posts here on HackerNews announcing a new model compression technique, your characterization above understates the technical innovations and practical impacts described in this MIT paper. Unlike traditional model compression work that simply applies existing techniques, SVDQuant synthesizes several ideas in a comprehensive new approach to model quantization: - Developing a novel outlier absorption mechanism using low-rank decomposition — this aspect alone seems quite novel, although the math is admittedly way beyond my level - Combining SVD with smoothing in a way that specifically addresses the unique challenges of diffusion models - Creating an innovative kernel fusion technique (they call it “Nunchaku”) that makes the theoretical benefits practically realizable, because without this, the extra computation required to implement the above steps would simply slow the model back down to baseline This isn't just incremental improvement - the paper achieves several breakthrough results: - First successful 4-bit quantization of both weights AND activations for diffusion models - 3.5x memory reduction for 12B parameter models while maintaining image quality - 3.0x speedup over existing 4-bit weight-only quantization approaches - Enables running 12B parameter models on consumer GPUs that previously couldn't handle them And, I’ll add, as someone who has been following the diffusion space quite actively for the last two years, the amount of creativity that can be unleashed when models are accessible to people with consumer GPUs is nothing short of astonishing. The authors took pains to validate their approach by testing it against three models (Flux, PixArt-Sigma, and SDXL) and along several quality-comparison axes (FID score, Image Reward, LPIPS, and PSNR). They also did a proper ablation study to see the contribution of each component in their approach to image quality. What particularly excites me about this paper is not the ability to run a model that eats 22GB of VRAM in just 7GB. The exciting thing is the prospect of running a 60GB model in 20GB of VRAM. I’m not sure whether anyone has or is planning to train such a monster, but I suspect that Midjourney, OpenAI, and Google all have significantly larger models running in their infrastructure than what can be run on consumer hardware. The more dimensions you can throw at image and video generation, the better things get.
- atlex2 2y agoSeriously nobody thought to use SVD on these weight matrices before?
- liuliu 2y agoI did try, but in a wrong way (try to SVD quantization error to recover quality (I.e. SVD(W - Q(W)))). The lightbulb moment in this paper is to do SVD on W and then quantize the remaining.
- gyrovagueGeist 2y agoThis problem seems like it would be very similar to the Low-Rank + Sparse decompositions that used to be popular in audio-visual filtering.