5 ms·

Running 27B dense model on M5 128GB is ok, but one can do better. On M5 128GB one can make use of the ram and use sparse MoE. For example, DeepSeek-V4-Flash wi

Running 27B dense model on M5 128GB is ok, but one can do better. On M5 128GB one can make use of the ram and use sparse MoE. For example, DeepSeek-V4-Flash will fit, served by DwarfStar (https://github.com/antirez/ds4 https://github.com/antirez/ds4). One will probably improve 2x the token/sec speed, given DS4F 13B activated params in the MoE are ~1/2 of the ~27B of the dense Qwen. 27B Of the Qwen fit even on a cheaper 24GB card, e.g. amd 7900xtx (<$1K?) or slightly dearer nvidia 3090 (with cuda). With ~900 GB/s bandwidth they will likely be ~50% faster than the M5 with 600 GB/s.

drnick1 3mo ago
Works beautifully on a 3090, very usable speed. Don't expect Opus 4.8-level performance, but there are some things you just need to keep local.
- ljosifov 3mo ago
  True - they are workhorses. Not super bright, but good enough for lots of everyday tasks. I've found sweet spot to be turning thinking off, as it adds small or no value, while increasing the token count and waiting time. Last 27B I used was https://huggingface.co/Jackrong/Qwopus3.6-27B-Coder-GGUF https://huggingface.co/Jackrong/Qwopus3.6-27B-Coder-GGUF - specifically post-train adapted a bit to run with thinking off. I saw today the 35B-A3B MoE from the same HF acc is out, downloading that rn to try.
  - kroaton 3mo ago
    Please don't use that garbage. Just use the base Qwen models or Nex/Orinth, as those are the only properly post-trained finetunes. The Qwopus models are marketing.
    - aand16 3mo ago
      Can you expand on why Qwopus is not recommended and what "Nex/Orinth" brings to the table?
    - ljosifov 3mo ago
      I usually doubt the 'small dataset tuned' variants. B/c ages ago (in the NN prehistory) I've done some NN training, and appreciate how hard it is to improve in general, and how easy it is to ruin a model in general while targeting a small dataset (LoRA-s are ok, that's different). That model/quant was the most recent one I was trying. But could not really use any of them, as the combo model + llama-server ground to a halt even at small context depth sizes on the amd gpu. Yesterday I finally found a good combo! So writing this for the benefit for anyone that may have the same h/w. Got around to /GOAL search for something better for the h/w (amd 7900xtx), and pi agent found a new best that actually seems it will be useful for real. As the 40 tok/s speed starts dropping only at 260K context depth?? Served by hipfire from this repo https://github.com/Kaden-Schutt/hipfire https://github.com/Kaden-Schutt/hipfire, that worked the best got on llama-benchy: | Context | Wall Time | PP (t/s) | TG (t/s) | |:-------:|:---------:|:--------:|:--------:| | 1,024 | 17.3s. | 660 | 40.8 | | 8,192 | 29.3s | 606 | 40.8 | | 32,768 | 70.5s | 599 | 40.7 | | 65,536 | 126.5s | 594 | 40.6 | | 131,072 | 235.5s | 591 | 39.0 | | 196,608 | 349.0s | 591 | 40.2 | | 260,000 | 451.3s | 594 | 37.2 | This is - hipfire daemon dflash_mode auto, serve qwen3.6:27b --kv-mode asym2. Models: hipfire pull qwen3.6:27b # 14 GB MQ4 quant, qwen3.6-27b.mq4 hipfire pull qwen3.6:27b-draft # 0.92 GB DFlash draft, qwen36-27b-dflash-mq4.hfq
brandall10 3mo ago
This is discussed in the article: "My personal impression is that within these quantizations Qwen 3.6 27B is as good as (or maybe slightly better than) DwarfStar4. Though, I won’t be surprised if for longer context projects DS4 has an edge."
- sfifs 3mo ago
  Used both. DeepSeek-4 Flash Q2 - last 6 layers Q4 quant with DwarfStar which just about fits in 128Gb is definitely superior IMO - my contexts tend to run typically 50-100k. Throughput tends to be about 12-13k tok/sec - just about acceptable.
kroaton 3mo ago
"DeepSeek-V4-Flash will fit" At Q2, 2bit? Lobotomized to death.
- ljosifov 3mo ago
  Hobbled - but not to death, the few times I use it (usually on a plane). I tried 2bit of a 20% REAP reduced experts. :-O That's the biggest that fits on my own h/w (3yrs old M2 Max 96gb). It's coherent, it does work, doesn't fall apart on casual use. IDK if better than dense 27b. Think 27b was slower on the same h/w. DS4F has got 1M context window. Nowadays with weeks long run hermes sessions, I get to 300k-400k context depths easily. The speed decline profile of DS4F with context depth increase is superior to any other model I try. (I try them all - love this stuff) Only previous model coming close on that is nemotron-cascade-2 (only 30b-a3b) - that also has 1M context window.

drnick1 3mo ago

Works beautifully on a 3090, very usable speed. Don't expect Opus 4.8-level performance, but there are some things you just need to keep local.

ljosifov 3mo ago
True - they are workhorses. Not super bright, but good enough for lots of everyday tasks. I've found sweet spot to be turning thinking off, as it adds small or no value, while increasing the token count and waiting time. Last 27B I used was https://huggingface.co/Jackrong/Qwopus3.6-27B-Coder-GGUF https://huggingface.co/Jackrong/Qwopus3.6-27B-Coder-GGUF - specifically post-train adapted a bit to run with thinking off. I saw today the 35B-A3B MoE from the same HF acc is out, downloading that rn to try.
- kroaton 3mo ago
  Please don't use that garbage. Just use the base Qwen models or Nex/Orinth, as those are the only properly post-trained finetunes. The Qwopus models are marketing.
  - aand16 3mo ago
    Can you expand on why Qwopus is not recommended and what "Nex/Orinth" brings to the table?
  - ljosifov 3mo ago
    I usually doubt the 'small dataset tuned' variants. B/c ages ago (in the NN prehistory) I've done some NN training, and appreciate how hard it is to improve in general, and how easy it is to ruin a model in general while targeting a small dataset (LoRA-s are ok, that's different). That model/quant was the most recent one I was trying. But could not really use any of them, as the combo model + llama-server ground to a halt even at small context depth sizes on the amd gpu. Yesterday I finally found a good combo! So writing this for the benefit for anyone that may have the same h/w. Got around to /GOAL search for something better for the h/w (amd 7900xtx), and pi agent found a new best that actually seems it will be useful for real. As the 40 tok/s speed starts dropping only at 260K context depth?? Served by hipfire from this repo https://github.com/Kaden-Schutt/hipfire https://github.com/Kaden-Schutt/hipfire, that worked the best got on llama-benchy: | Context | Wall Time | PP (t/s) | TG (t/s) | |:-------:|:---------:|:--------:|:--------:| | 1,024 | 17.3s. | 660 | 40.8 | | 8,192 | 29.3s | 606 | 40.8 | | 32,768 | 70.5s | 599 | 40.7 | | 65,536 | 126.5s | 594 | 40.6 | | 131,072 | 235.5s | 591 | 39.0 | | 196,608 | 349.0s | 591 | 40.2 | | 260,000 | 451.3s | 594 | 37.2 | This is - hipfire daemon dflash_mode auto, serve qwen3.6:27b --kv-mode asym2. Models: hipfire pull qwen3.6:27b # 14 GB MQ4 quant, qwen3.6-27b.mq4 hipfire pull qwen3.6:27b-draft # 0.92 GB DFlash draft, qwen36-27b-dflash-mq4.hfq

brandall10 3mo ago

This is discussed in the article: "My personal impression is that within these quantizations Qwen 3.6 27B is as good as (or maybe slightly better than) DwarfStar4. Though, I won’t be surprised if for longer context projects DS4 has an edge."

sfifs 3mo ago
Used both. DeepSeek-4 Flash Q2 - last 6 layers Q4 quant with DwarfStar which just about fits in 128Gb is definitely superior IMO - my contexts tend to run typically 50-100k. Throughput tends to be about 12-13k tok/sec - just about acceptable.

kroaton 3mo ago

"DeepSeek-V4-Flash will fit" At Q2, 2bit? Lobotomized to death.

ljosifov 3mo ago
Hobbled - but not to death, the few times I use it (usually on a plane). I tried 2bit of a 20% REAP reduced experts. :-O That's the biggest that fits on my own h/w (3yrs old M2 Max 96gb). It's coherent, it does work, doesn't fall apart on casual use. IDK if better than dense 27b. Think 27b was slower on the same h/w. DS4F has got 1M context window. Nowadays with weeks long run hermes sessions, I get to 300k-400k context depths easily. The speed decline profile of DS4F with context depth increase is superior to any other model I try. (I try them all - love this stuff) Only previous model coming close on that is nemotron-cascade-2 (only 30b-a3b) - that also has 1M context window.