4 ms·
Yes, I believe in theory a correctly written program could scale across sockets, depending on the problem at hand. But where is your data? For llama.cpp? For w
by buildbot 2y ago
Yes, I believe in theory a correctly written program could scale across sockets, depending on the problem at hand.
But where is your data? For llama.cpp? For whatever dual socket CPU system you want. That’s all I am claiming.
- adrian_b 2y agoGoogling for what you ask has found immediately this discussion: https://github.com/ggml-org/llama.cpp/discussions/11733 https://github.com/ggml-org/llama.cpp/discussions/11733 about the scaling of llama.cpp and DeepSeek on some dual-socket AMD systems. While it was rather tricky, after many experiments they have obtained an almost double speed on two sockets, especially on AMD Turin. However, if you look at the actual benchmark data, that must be much lower than what is really possible, because their test AMD Turin system (named there P1) had only two thirds of the memory channels populated, i.e. performance limited by memory bandwidth could be increased by 50%, and they had 16-core CPUs, so performance limited by computation could be increased around 10 times.
- buildbot 2y agoCool, I didn’t find that one! Thanks. A single 192 core Epyc is 11k by itself, so I’d probably go for the simpler integrated M3 ultra solution…