3 ms·
What's your hardware? I've got 64GB RAM and a 4090 here, wondering if it's worth a play.
by regularfry 3mo ago
What's your hardware? I've got 64GB RAM and a 4090 here, wondering if it's worth a play.
- andy99 3mo agoIt’s the Strix halo (AMD) with 128 GB shared memory. The 4bit quant is ~75GB. Unfortunately I don’t know about the best way of running on an Nvidia gpu, you could try llama.cpp and offloading as many layers as possible into the gpu and using RAM for the rest, not sure if that would slow it down too much though.
- regularfry 3mo agoI got usable token rates (10-20tps from memory, so marginal) with the Qwen A10B a while back, well before all the new speculative speedups landed in llama.cpp. There's an unmerged branch which allegedly supports this, but I don't know how well yet. Looks worth investigating but I might give it a few days to see what bugs get shaken loose.