3 ms·
What kind of machine do you have in your homelab that can run this model?!
by looksjjhg 1mo ago
What kind of machine do you have in your homelab that can run this model?!
- petu 1mo agoThis is needs ~80GB of fast memory at 4 bits per weight. Faster memory is better, but probably even something like 3090 + 64GB RAM should work (not fast, but maybe even 20-30 t/s? llama.cpp support pending).
- lnenad 1mo agoI've got a 48c Epyc with 2x3090s and 512gb ddr4 3200. It's good enough for 25+ tps with deepseek so I'm hoping for similar performance with less overthinking.