3 ms·
Yes, it can be done. I'm running a 24-channel DDR5 dual-EPYC rig and get good speed on large MoE models. I only use the GPU for context processing. They're act
by DrPhish 2y ago
Yes, it can be done. I'm running a 24-channel DDR5 dual-EPYC rig and get good speed on large MoE models. I only use the GPU for context processing.
They're actually a best-case for CPU inference vs dense models. I usually run deepseek 2.5 quanted to q8, but if this model works well I'll probably switch to it once support hits llama.cpp.
- Tepix 2y agoInteresting, what RAM do you use exactly? 24x 16GB DDR5-6000 DIMMs? It seems that those boards only support up to DDR5-4800: https://geizhals.de/?cat=mbsp3&xf=4921_2%7E493_24x+DDR5+DIMM%7E644_SP5 https://geizhals.de/?cat=mbsp3&xf=4921_2%7E493_24x+DDR5+DIMM... Does the core count matter or can you get away with the smallest 2x EPYC 9015 configuration? What are "good speeds"?
- Tepix 2y agoLooks like we will soon get boards supporting 24x DDR5-6000 for the EPYC 9005 CPUs.
- DrPhish 2y agoI use 24 sticks of ddr5-4800, which gets me up to 9t/s on deepseek 2.5 at q8. 48 threads was optimal in llama.cpp. I would like to move to epyc 9005 chips and ddr5-6000, but it is cost prohibitive with CPUs still over $10k each on eBay. I followed the guide at https://rentry.co/miqumaxx/ https://rentry.co/miqumaxx/
- Tepix 2y agoHow many cores do your CPUs have? Are you using the 64 core EPYC 9334 mentioned in the linked page? Do that many cores provide a speedup versus having fewer cores?
- DrPhish 2y agoYes I have the same engineering sample. I can only keep 48 cores fed with max memory bandwidth given current llama.cpp architectural constraints
- ConnectX-2 2y ago>I only use the GPU for context processing. If your GPU has enough VRAM to support it, you might benefit from https://github.com/kvcache-ai/ktransformers https://github.com/kvcache-ai/ktransformers