5 ms·
Interesting, what RAM do you use exactly? 24x 16GB DDR5-6000 DIMMs? It seems that those boards only support up to DDR5-4800: https://geizhals.de/?cat=mbsp3&xf=4
by Tepix 2y ago
Interesting, what RAM do you use exactly? 24x 16GB DDR5-6000 DIMMs? It seems that those boards only support up to DDR5-4800: https://geizhals.de/?cat=mbsp3&xf=4921_2%7E493_24x+DDR5+DIMM%7E644_SP5 https://geizhals.de/?cat=mbsp3&xf=4921_2%7E493_24x+DDR5+DIMM...
Does the core count matter or can you get away with the smallest 2x EPYC 9015 configuration?
What are "good speeds"?
- Tepix 2y agoLooks like we will soon get boards supporting 24x DDR5-6000 for the EPYC 9005 CPUs.
- DrPhish 2y agoI use 24 sticks of ddr5-4800, which gets me up to 9t/s on deepseek 2.5 at q8. 48 threads was optimal in llama.cpp. I would like to move to epyc 9005 chips and ddr5-6000, but it is cost prohibitive with CPUs still over $10k each on eBay. I followed the guide at https://rentry.co/miqumaxx/ https://rentry.co/miqumaxx/
- Tepix 2y agoHow many cores do your CPUs have? Are you using the 64 core EPYC 9334 mentioned in the linked page? Do that many cores provide a speedup versus having fewer cores?
- DrPhish 2y agoYes I have the same engineering sample. I can only keep 48 cores fed with max memory bandwidth given current llama.cpp architectural constraints