5 ms·
It’s 400B but it’s mixture of experts so how many are active at any time?
by cj00 6mo ago
It’s 400B but it’s mixture of experts so how many are active at any time?
- simonw 6mo agoLooks like it's Qwen3.5-397B-A17B so 17B active. https://github.com/Anemll/flash-moe/tree/iOS-App https://github.com/Anemll/flash-moe/tree/iOS-App
- stingraycharles 6mo agoOne expert is 17B, but more than one expert can be active at any time. I believe it’s actually more like 80B active.
- zozbot234 6mo agoI don't think this is correct, "active parameters" is quite unambiguous in that it means a sum of all active experts plus shared parameters.
- fouc 6mo agolooks like they meant “effective dense size” which is the square root of total params×active params, so in this case sqrt(397 x 17) = ~82
- zozbot234 6mo agoBut the claim that "one expert is 17B" is incorrect. Experts are picked with per-layer granularity (expert 1 for layer X may well be entirely unrelated to expert 1 for layer Y), and the individual layer-experts are tiny. The writeup for the original experiment is very clear on this.
- stingraycharles 6mo agoOk I am by no means an expert on this and I immediately stand corrected. But as I understand it, in order to understand the amount of active memory that’s required, it’s more accurate to go by the ~82B number, right?
- zozbot234 6mo agoThe ~82B figure is an attempt to compare performance to an equivalent dense model. The amount of active parameters is given by the ~17B.
- thecopy 6mo agoStupid question: can i run this on my 64GB/1TB mac somehow easily? Or this requires custom coding? 4bit is ~200GB EDIT: found this in the replies: https://github.com/Anemll/flash-moe/tree/iOS-App https://github.com/Anemll/flash-moe/tree/iOS-App
- jnovek 6mo agoI have a 64G/1T Studio with an M1 Ultra. You can probably run this model to say you’ve done it but it wouldn’t be very practical. Also I wouldn’t trust 3-bit quantization for anything real. I run a 5-bit qwen3.5-35b-A3B MoE model on my studio for coding tasks and even the 4-bit quant was more flaky (hallucinations, and sometimes it would think about running tools calls and just not run them, lol). If you decided to give it a go make sure to use the MLX over the GGUF version! You’ll get a bit more speed out of it.
- Aurornis 6mo agoRunning larger-than-RAM LLMs is an interesting trick, but it's not practical. The output would be extremely slow and your computer would be burning a lot of power to get there. The heavy quantizations and other tricks (like reducing the number of active experts) used in these demos severely degrade the quality. With 64GB of RAM you should look into Qwen3.5-27B or Qwen3.5-35B-A3B. I suggest Q5 quantization at most from my experience. Q4 works on short responses but gets weird in longer conversations.
- freedomben 6mo agoI've tried a number of experiments, and agree completely. If it doesn't fit in RAM, it's so slow as to be impractical and almost useless. If you're running things overnight, then maybe, but expect to wait a very long time for any answers.
- zozbot234 6mo agoCurrent local-AI frameworks do a bad job of supporting the doesn't-fit-in-RAM case, though. Especially when running combined CPU+GPU inference. If you aren't very careful about how you run these experiments, the framework loads all weights from disk into RAM only for the OS to swap them all out (instead of mmap-ing the weights in from an existing file, or doing something morally equivalent as with the original MacBook Pro experiment) which is quite wasteful! This approach also makes less sense for discrete GPUs where VRAM is quite fast but scarce, and the GPU's PCIe link is a key bottleneck. I suppose it starts to make sense again once you're running the expert layers with CPU+RAM.
- Hasslequest 6mo agoStill pretty good considering 17B is what one would run on a 16GB laptop at Q6 with reasonable headroom
- anshumankmr 6mo agoAren't most companies doing MoE at this point?
- butILoveLife 6mo ago[dead]