4 ms·

No... this headline is incorrect. You can't do that. I think they've confused the performance of running one of the small distills to existing smaller models. T

No... this headline is incorrect. You can't do that. I think they've confused the performance of running one of the small distills to existing smaller models. Two Arc cards cannot fit a 4 bit k-quant of a 671b model. But a portable (no install) way to run llama.cpp on intel GPUs is really cool.

rgbrgb 2y ago
yep, title is inaccurate. it's a distill into Qwen 7B DeepSeek-R1-Distill-Qwen-7B-Q4_K_M.gguf
- zamadatix 2y ago
  The document contains multiple sections. The initial section does reference DeepSeek-R1-Distill-Qwen-7B-Q4_K_M.gguf as the example model but if you continue reading further you'll see a section referencing running DeepSeek-R1-Q4_K_M.gguf plus claims several other variations have been tested. It's a bit less exciting when you see they're just talking about offloading parts from the large amount of DRAM.
  - genewitch 2y ago
    So you thought there was some magical way to get >600B parameters in a couple of GPUs? Also, LM studio lets you run smaller models in front of larger ones, so I could see having a few GPU in front really speeding up using R1 for inference.
    - hmottestad 2y ago
      The MoE architecture allows you to keep the entire active model on a single GPU. If two consecutive tokens use the same export then the second token is going to be much faster.
      - utopcell 2y ago
        What is the probability of that happening?
        zamadatix 2y ago
        DeepSeek V3/R1 uses 8 routed experts out of 256, so not all as often as one would like. That said, having even just a single GPU will greatly speed up prompt processing which is worth it even if the inference speed was the same. Ktransformers has a document about using CPU + a single 4090D to reach decent tokens/s but I'm not sure how much of the perf is due to the 4090D vs other optimizations/changes for the CPU side https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/DeepseekR1_V3_tutorial.md https://github.com/kvcache-ai/ktransformers/blob/main/doc/en... The final step of going to 6 experts instead of 8 feels like cheating (not a lossless optimization).
        genewitch 2y ago
        where does 256 come from? it's repeated in here and elsewhere that a single expert is 37B sized, so you'd have to have way more than "several hundred billion parameters", to hold 256 of those? Maybe i don't understand the architecture, but if that's the case, then everyone repeating 37B doesn't, either.
        zamadatix 2y ago
        I think this diagram from the DeepSeekMoE paper explains it the clearest: https://i.imgur.com/CRKttob.png https://i.imgur.com/CRKttob.png The one on the right is how the feed forward layers of DeepSeek V3/R1 work, blue and green are experts, and everything in that right section is what counts as "active parameters". K (K=8 for these models, but you can customize that if you want) experts of 256 per layer are activated at a time. The 256 comes from the model file, it's just how many they chose to build it with. In these models there is also 1 shared expert which is always active in the layer. The router picks which k routed experts to use each forward pass and then a gating mechanism combines the outputs. If you sum the 1 shared expert + K routed experts + router + output networks you end up with 37 B parameters active for each feed forward layer pass. The individual experts are therefore much smaller than the total (probably something like 4 B parameters each? I've never really checked that directly). Or, for the short answer: "37 B is the active parameters of 9 experts + 'overhead', not the parameters of a single expert".
    - zamadatix 2y ago
      I had also initially assumed the title was supposed to reference something new about running a distilled variant as well. When I finished reading through and found out the news was just that you can also do this sort of "split" setup with Intel gear too it removed any further hope of excitement. DeepSeek employs multi-token prediction which enables self-speculative decoding without needing to employ a separate draft model. Or at least that's what I understood the value of multi-token prediction to be.
- colorant 2y ago
  See this section https://github.com/intel/ipex-llm/blob/main/docs/mddocs/Quickstart/llamacpp_portable_zip_gpu_quickstart.md#flashmoe-for-deepseek-v3r1 https://github.com/intel/ipex-llm/blob/main/docs/mddocs/Quic...
Cheer2171 2y ago
You don't have to go that far down the page to see it is paging to system RAM: Requirements: 380GB CPU Memory 1-8 ARC A770 500GB Disk
- superkuh 2y ago
  Yep. That's why the headline is incorrect. 380GB of the model on CPU system RAM and 32GB on some ARC GPUs. The ratio, 380/32, is obvious. Most of the processing is being done on the CPU. The GPU are little bit icing in this context. Fast, sure, but having to wait for the CPU layers (that's how layer splits work with llama.cpp). I think changing the end of headline to "Xeon w/380GB RAM" would stop it from being incorrect and misleading.
  - Cheer2171 2y ago
    "with" does not mean "entirely on" Edit: but what you added in your edit is right, it would be more accurate to append the system ram requirement
  - ryao 2y ago
    What if it does not need to read from system RAM for every token by reusing experts whenever they just happen to be in VRAM from being used for the previous token? If the selected experts do not change often, this is doable on paper.
    - hmottestad 2y ago
      That’s probably the main performance benefit of using the GPU. If you’re changing the active expert for every single token then it wouldn’t be any faster than just running it on the CPU. Once you can reuse the active expert for two tokens you’re already going to be a lot faster than just the CPU. More GPUs let you keep more experts active at a time.
    - hexaga 2y ago
      Expert distribution should be approximately random token-by-token, so not likely.
deleted 2y ago
[deleted]
ryao 2y ago
It is theoretically possible. Each token only needs 37B parameters and if the same experts are chosen often, it would behave closer to a 37B model than a 671B model, since reusing experts can skip loads from system RAM. You might still be right since I have not confirmed that the selected experts change infrequently doing prompt processing / token generation, and someone could have botched the headline. However, treating Deepseek like llama 3 when reasoning about VRAM requirements is not necessarily correct.
- superkuh 2y ago
  MoE is pretty enabling after you've spent all the extra $$$$ to stuff your server CPU memory channels with ram so it's possible to run at all. But it's still spending a lot of money which makes this a lot less novel or interesting than "just on 1~2 Arc A770" implies. Especially for the marginal performance that even 8-12 channels of CPU memory bandwidth gets you.
  - utopcell 2y ago
    Is this amount of RAM really that expensive? 6x 64GiB DDR4 DIMMs are < $1,000.
  - utopcell 2y ago
    Actually, 384GiB is already <$400 [1]. [1] https://www.amazon.com/NEMIX-RAM-DDR4-2666MHz-PC4-21300-Reduced/dp/B0BJFN1JYS https://www.amazon.com/NEMIX-RAM-DDR4-2666MHz-PC4-21300-Redu...
    - superkuh 2y ago
      A slow-end DDR4 speed older generation Xeon system is unlikely to be used by Intel for this benchmark. It's far more likely they used an expensive DDR5 modern Xeon with as many memory channels as they could. Single user LLM inference is memory bandwidth bottlenecked. I just can't see Intel using old/deprecated hardware. And if someone not Intel were to build a Xeon DDR4 system it wouldn't reach the DDR5 tokens/s speeds reported here. The reason they used a Xeon is memory channels. Non-server CPUs only have 2 but modern Xeons have 8 to 12 depending on generation/type. And the Xeons with the most are the most $$$$ and it ends up cheaper to just get a GPU or dedicated accelerator.

rgbrgb 2y ago

yep, title is inaccurate. it's a distill into Qwen 7B DeepSeek-R1-Distill-Qwen-7B-Q4_K_M.gguf

zamadatix 2y ago
The document contains multiple sections. The initial section does reference DeepSeek-R1-Distill-Qwen-7B-Q4_K_M.gguf as the example model but if you continue reading further you'll see a section referencing running DeepSeek-R1-Q4_K_M.gguf plus claims several other variations have been tested. It's a bit less exciting when you see they're just talking about offloading parts from the large amount of DRAM.
- genewitch 2y ago
  So you thought there was some magical way to get >600B parameters in a couple of GPUs? Also, LM studio lets you run smaller models in front of larger ones, so I could see having a few GPU in front really speeding up using R1 for inference.
  - hmottestad 2y ago
    The MoE architecture allows you to keep the entire active model on a single GPU. If two consecutive tokens use the same export then the second token is going to be much faster.
    - utopcell 2y ago
      What is the probability of that happening?
      - zamadatix 2y ago
        DeepSeek V3/R1 uses 8 routed experts out of 256, so not all as often as one would like. That said, having even just a single GPU will greatly speed up prompt processing which is worth it even if the inference speed was the same. Ktransformers has a document about using CPU + a single 4090D to reach decent tokens/s but I'm not sure how much of the perf is due to the 4090D vs other optimizations/changes for the CPU side https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/DeepseekR1_V3_tutorial.md https://github.com/kvcache-ai/ktransformers/blob/main/doc/en... The final step of going to 6 experts instead of 8 feels like cheating (not a lossless optimization).
        genewitch 2y ago
        where does 256 come from? it's repeated in here and elsewhere that a single expert is 37B sized, so you'd have to have way more than "several hundred billion parameters", to hold 256 of those? Maybe i don't understand the architecture, but if that's the case, then everyone repeating 37B doesn't, either.
        zamadatix 2y ago
        I think this diagram from the DeepSeekMoE paper explains it the clearest: https://i.imgur.com/CRKttob.png https://i.imgur.com/CRKttob.png The one on the right is how the feed forward layers of DeepSeek V3/R1 work, blue and green are experts, and everything in that right section is what counts as "active parameters". K (K=8 for these models, but you can customize that if you want) experts of 256 per layer are activated at a time. The 256 comes from the model file, it's just how many they chose to build it with. In these models there is also 1 shared expert which is always active in the layer. The router picks which k routed experts to use each forward pass and then a gating mechanism combines the outputs. If you sum the 1 shared expert + K routed experts + router + output networks you end up with 37 B parameters active for each feed forward layer pass. The individual experts are therefore much smaller than the total (probably something like 4 B parameters each? I've never really checked that directly). Or, for the short answer: "37 B is the active parameters of 9 experts + 'overhead', not the parameters of a single expert".
  - zamadatix 2y ago
    I had also initially assumed the title was supposed to reference something new about running a distilled variant as well. When I finished reading through and found out the news was just that you can also do this sort of "split" setup with Intel gear too it removed any further hope of excitement. DeepSeek employs multi-token prediction which enables self-speculative decoding without needing to employ a separate draft model. Or at least that's what I understood the value of multi-token prediction to be.
colorant 2y ago
See this section https://github.com/intel/ipex-llm/blob/main/docs/mddocs/Quickstart/llamacpp_portable_zip_gpu_quickstart.md#flashmoe-for-deepseek-v3r1 https://github.com/intel/ipex-llm/blob/main/docs/mddocs/Quic...

Cheer2171 2y ago

You don't have to go that far down the page to see it is paging to system RAM: Requirements: 380GB CPU Memory 1-8 ARC A770 500GB Disk

superkuh 2y ago
Yep. That's why the headline is incorrect. 380GB of the model on CPU system RAM and 32GB on some ARC GPUs. The ratio, 380/32, is obvious. Most of the processing is being done on the CPU. The GPU are little bit icing in this context. Fast, sure, but having to wait for the CPU layers (that's how layer splits work with llama.cpp). I think changing the end of headline to "Xeon w/380GB RAM" would stop it from being incorrect and misleading.
- Cheer2171 2y ago
  "with" does not mean "entirely on" Edit: but what you added in your edit is right, it would be more accurate to append the system ram requirement
- ryao 2y ago
  What if it does not need to read from system RAM for every token by reusing experts whenever they just happen to be in VRAM from being used for the previous token? If the selected experts do not change often, this is doable on paper.
  - hmottestad 2y ago
    That’s probably the main performance benefit of using the GPU. If you’re changing the active expert for every single token then it wouldn’t be any faster than just running it on the CPU. Once you can reuse the active expert for two tokens you’re already going to be a lot faster than just the CPU. More GPUs let you keep more experts active at a time.
  - hexaga 2y ago
    Expert distribution should be approximately random token-by-token, so not likely.

deleted 2y ago

[deleted]

ryao 2y ago

It is theoretically possible. Each token only needs 37B parameters and if the same experts are chosen often, it would behave closer to a 37B model than a 671B model, since reusing experts can skip loads from system RAM. You might still be right since I have not confirmed that the selected experts change infrequently doing prompt processing / token generation, and someone could have botched the headline. However, treating Deepseek like llama 3 when reasoning about VRAM requirements is not necessarily correct.

superkuh 2y ago
MoE is pretty enabling after you've spent all the extra $$$$ to stuff your server CPU memory channels with ram so it's possible to run at all. But it's still spending a lot of money which makes this a lot less novel or interesting than "just on 1~2 Arc A770" implies. Especially for the marginal performance that even 8-12 channels of CPU memory bandwidth gets you.
- utopcell 2y ago
  Is this amount of RAM really that expensive? 6x 64GiB DDR4 DIMMs are < $1,000.
- utopcell 2y ago
  Actually, 384GiB is already <$400 [1]. [1] https://www.amazon.com/NEMIX-RAM-DDR4-2666MHz-PC4-21300-Reduced/dp/B0BJFN1JYS https://www.amazon.com/NEMIX-RAM-DDR4-2666MHz-PC4-21300-Redu...
  - superkuh 2y ago
    A slow-end DDR4 speed older generation Xeon system is unlikely to be used by Intel for this benchmark. It's far more likely they used an expensive DDR5 modern Xeon with as many memory channels as they could. Single user LLM inference is memory bandwidth bottlenecked. I just can't see Intel using old/deprecated hardware. And if someone not Intel were to build a Xeon DDR4 system it wouldn't reach the DDR5 tokens/s speeds reported here. The reason they used a Xeon is memory channels. Non-server CPUs only have 2 but modern Xeons have 8 to 12 depending on generation/type. And the Xeons with the most are the most $$$$ and it ends up cheaper to just get a GPU or dedicated accelerator.