4 ms·

CPU-only is, very unfortunately, infeasible for reasoning models. This setup would be great for deepseek v3 or (more fittingly) the 405B llama 3.1 model, but 6

CPU-only is, very unfortunately, infeasible for reasoning models. This setup would be great for deepseek v3 or (more fittingly) the 405B llama 3.1 model, but 6-7 tokens per second on a reasoning model is 100% getting (well) into seconds-per-token territory if you consider only the final answer. (You don’t have to take it from me: if CPU were good enough, AMD’s valuation would be 100x its current value.)

samvher 2y ago
Given what we just saw in terms of the DeepSeek team squeezing a lot of extra performance out of more efficient implementation on GPU, and the model still being optimized for GPU rather than CPU - is it unreasonable to think that in the $6k setup described, some performance might still be left on the table that could be squeezed out with some better optimization for these particular CPUs?
- telotortium 2y ago
  Maybe a little, but FLOPs and memory bandwidth don't lie.
- snovv_crash 2y ago
  No, because the bottleneck is RAM bandwidth. This is already quantized and otherwise is essentially random so can't be compressed in any meaningful way.
  - menaerus 2y ago
    How much bandwidth do we actually need per-token generation? Let's take one open-source model as a starting point since not all models are created the same.
    - ryao 2y ago
      How big are the active weights? That how much bandwidth you need per second per token.
    - snovv_crash 2y ago
      For non-MoE models, it needs to flow the entire model through the CPU. So if it is a 32B parameter model quantised to 8b/parameter, that is 32GB of RAM bandwidth per token. If your RAM does 64GB/s that is 2 tok/s.
      - menaerus 2y ago
        I didn't get the impression that the math around it is that simplistic. The first obvious reason I can think of now is the attention mechanism being used. Both GQA and MQA demand less compute and therefore less bandwidth than MHA.
- ryao 2y ago
  The answer to your question is yes. There is an open issue with llama.cpp about this very thing: https://github.com/ggerganov/llama.cpp/issues/11333 https://github.com/ggerganov/llama.cpp/issues/11333 The TLDR is that llama.cpp’s NUMA support is suboptimal, which is hurting performance versus what it should be on this machine. A single socket version likely would perform better until it is fixed. After it is fixed, a dual socket machine would likely run at the same speed as a single socket machine. If someone implemented a GEMV that scales with NUMA nodes (i.e. PBLAS, but for the data types used in inference), it might be possible to get higher performance from a dual socket machine than we get from a single socket machine.
qingcharles 2y ago
It honestly depends on your use case. I often run bigger, slower models on my PC and let them just tootle along in the background grinding out their response while I work on something else.
- grahamj 2y ago
  Yeah this is what I was thinking, or maybe use smaller models to work on the prompt then fire it off to the biggie while you do something else.
brandall10 2y ago
"This setup would be great for deepseek v3 or (more fittingly) the 405B llama 3.1 model" v3 yes w/ 37B activated params, yes, but terrible on 405B as it's a dense model.

samvher 2y ago

Given what we just saw in terms of the DeepSeek team squeezing a lot of extra performance out of more efficient implementation on GPU, and the model still being optimized for GPU rather than CPU - is it unreasonable to think that in the $6k setup described, some performance might still be left on the table that could be squeezed out with some better optimization for these particular CPUs?

telotortium 2y ago
Maybe a little, but FLOPs and memory bandwidth don't lie.
snovv_crash 2y ago
No, because the bottleneck is RAM bandwidth. This is already quantized and otherwise is essentially random so can't be compressed in any meaningful way.
- menaerus 2y ago
  How much bandwidth do we actually need per-token generation? Let's take one open-source model as a starting point since not all models are created the same.
  - ryao 2y ago
    How big are the active weights? That how much bandwidth you need per second per token.
  - snovv_crash 2y ago
    For non-MoE models, it needs to flow the entire model through the CPU. So if it is a 32B parameter model quantised to 8b/parameter, that is 32GB of RAM bandwidth per token. If your RAM does 64GB/s that is 2 tok/s.
    - menaerus 2y ago
      I didn't get the impression that the math around it is that simplistic. The first obvious reason I can think of now is the attention mechanism being used. Both GQA and MQA demand less compute and therefore less bandwidth than MHA.
ryao 2y ago
The answer to your question is yes. There is an open issue with llama.cpp about this very thing: https://github.com/ggerganov/llama.cpp/issues/11333 https://github.com/ggerganov/llama.cpp/issues/11333 The TLDR is that llama.cpp’s NUMA support is suboptimal, which is hurting performance versus what it should be on this machine. A single socket version likely would perform better until it is fixed. After it is fixed, a dual socket machine would likely run at the same speed as a single socket machine. If someone implemented a GEMV that scales with NUMA nodes (i.e. PBLAS, but for the data types used in inference), it might be possible to get higher performance from a dual socket machine than we get from a single socket machine.

qingcharles 2y ago

It honestly depends on your use case. I often run bigger, slower models on my PC and let them just tootle along in the background grinding out their response while I work on something else.

grahamj 2y ago
Yeah this is what I was thinking, or maybe use smaller models to work on the prompt then fire it off to the biggie while you do something else.

brandall10 2y ago

"This setup would be great for deepseek v3 or (more fittingly) the 405B llama 3.1 model" v3 yes w/ 37B activated params, yes, but terrible on 405B as it's a dense model.