3 ms·
What can't you run? Unquantised large text models are the only thing I can't run Stable diffusion, stable video, text models, audio models, I never had issues
by jokethrowaway 3y ago
What can't you run? Unquantised large text models are the only thing I can't run
Stable diffusion, stable video, text models, audio models, I never had issues with anything yet
- michaelt 3y agoThe 4090 is in a bit of a funny space for LLMs. There's a lot of open weights activity around 7B/13B models which the 4090 will run with ease. But you could can run those OK on much cheaper cards like the 4070Ti (which is of course why they're popular). And there's a lot of open weights activity around 70B and 8x7B models which are state-of-the-art - but too big to fit on a 4090. There's not much activity around 30B models, which are too big to be mainstream and too small to be cutting edge. If you're specifically looking to QLoRA fine-tune a 7B/13B model a 4090 can do that - but if you want to go bigger than that you'll end up using a cloud multi-gpu machine anyway.