3 ms·
But you can get one of these quantized models to run effectively on a 3090? If so, I'd love detailed instructions. The guide you posted earlier goes over my (
by sleight42 1y ago
But you can get one of these quantized models to run effectively on a 3090?
If so, I'd love detailed instructions.
The guide you posted earlier goes over my (and likely many others') head!
- danielhanchen 1y agoOh yes definitely! Oh wait is the guide too long / wordy? This section https://docs.unsloth.ai/basics/qwen3-coder-how-to-run-locally#llama.cpp-run-qwen3-tutorial https://docs.unsloth.ai/basics/qwen3-coder-how-to-run-locall... shows how to run it on a 3090
- sleight42 1y agoKind of you to respond! Thanks! I have pretty bad ADHD. And I've only run locally using kobold; dilettante at DIY AI. So, yeah, I'm a bit lost in it.
- danielhanchen 1y agoOh sorry - for Kobold - I think it uses llama.cpp behind the hood? I think Kobold has some guides on using custom GGUFs