2 ms·
All the "consumer grade GPUs" terminology makes it seem like you could run it on a variety of models, but like so many of these posts, is this a 4090 exclusive?
by causality0 3y ago
All the "consumer grade GPUs" terminology makes it seem like you could run it on a variety of models, but like so many of these posts, is this a 4090 exclusive?
- int_19h 3y agoI can't think of anything that is a 4090 exclusive. What usually matters is VRAM, so if something needs 24Gb, then 3090 is also an option, or dual 12Gb cards. Anyway, the technique that they describe is a general one that should broadly improve the ability to run larger models on smaller GPUs by drastically improving perf for CPU offloading. They demonstrate it using both 4090 running the largest models at fp16, but also 2080Ti running the same 4-bit quantized, and they still get ~3x speedup for LLaMA. So this very much sounds like it'll make 33B models the new default on the desktops, while people with even a single 3090 or 4090 will now be able to run 70B at realtime chat speeds.