3 ms·
I use 24 sticks of ddr5-4800, which gets me up to 9t/s on deepseek 2.5 at q8. 48 threads was optimal in llama.cpp. I would like to move to epyc 9005 chips and d
by DrPhish 2y ago
I use 24 sticks of ddr5-4800, which gets me up to 9t/s on deepseek 2.5 at q8. 48 threads was optimal in llama.cpp. I would like to move to epyc 9005 chips and ddr5-6000, but it is cost prohibitive with CPUs still over $10k each on eBay.
I followed the guide at https://rentry.co/miqumaxx/ https://rentry.co/miqumaxx/
- Tepix 2y agoHow many cores do your CPUs have? Are you using the 64 core EPYC 9334 mentioned in the linked page? Do that many cores provide a speedup versus having fewer cores?
- DrPhish 2y agoYes I have the same engineering sample. I can only keep 48 cores fed with max memory bandwidth given current llama.cpp architectural constraints