3 ms·
> If you try to do any kind of agentic coding locally, where contexts are 7k+, local hardware completely falls over. With my 5070 Ti + 2080 Ti I have Qwen 3 Co
by magicalhippo 1y ago
> If you try to do any kind of agentic coding locally, where contexts are 7k+, local hardware completely falls over.
With my 5070 Ti + 2080 Ti I have Qwen 3 Coder 30B Q4_K_M running entirely on the GPUs with 16k context. Not great for larger code bases, but not nothing either.
Asking it to summarize llama-model-loader.cpp, which is about ~12k tokens, the TTFT is ~13 seconds and generation speed is about 55 tok/sec.
So yeah, for local stuff it's pick any two of large models, long contexts and decent speed.
- linuxftw 1y agoYeah, that sounds decent for some one-shots. The unified memory systems can have longer back-and-forth context chats, but at slower speed (at least on AMD). I find Qwen 3 Coder to be quite usable, I get around 20TPS on my AMD AI 350 system, as long as the net-new context isn't too big.
- magicalhippo 1y agoI was looking at the 5060 Ti 16GB, and it has about half the memory bandwidth of the 5070 Ti, but at half the price here. With four of them you'd have 64 GB VRAM and still a lot cheaper than a 5090. Should get around 20-25 TPS for Qwen 3 Coder 30B, which is within usable range. Need a big case tho or go bitcoin miner style. Not seriously thinking about it, just playing around.