3 ms·
It's a MoE model and the A3B stands for 3 Billion active parameters, like the recent Gemma 4. You can try to offload the experts on CPU with llama.cpp (--cpu-m
by mhitza 6mo ago
It's a MoE model and the A3B stands for 3 Billion active parameters, like the recent Gemma 4.
You can try to offload the experts on CPU with llama.cpp (--cpu-moe) and that should give you quite the extra context space, at a lower token generation speed.
- dgb23 6mo agoDo I expect the same memory footprint from an N active parameters as from simply N total parameters?
- deleted 6mo ago[deleted]
- daemonologist 6mo agoNo - this model has the weights memory footprint of a 35B model (you do save a little bit on the KV cache, which will be smaller than the total size suggests). The lower number of active parameters gives you faster inference, including lower memory bandwidth utilization, which makes it viable to offload the weights for the experts onto slower memory. On a Mac, with unified memory, this doesn't really help you. (Unless you want to offload to nonvolatile storage, but it would still be painfully slow.) All that said you could probably squeeze it onto a 36GB Mac. A lot of people run this size model on 24GB GPUs, at 4-5 bits per weight quantization and maybe with reduced context size.
- pdyc 6mo agoi dont get it, mac has unified memory how would offloading experts to cpu help?
- bee_rider 6mo agoI bet the poster just didn’t remember that important detail about Macs, it is kind of unusual from a normal computer point of view. I wonder though, do Macs have swap, coupled unused experts be offloaded to swap?
- abhikul0 6mo agoOf course the swap is there for fallback but I hate using it lol as I don't want to degrade SSD longevity.
- abhikul0 6mo agoMac has unified memory, so 36GB is 36GB for everything- gpu,cpu.
- mhitza 6mo agoFor sure I was running on autopilot with that reply. Though in Q4 I would expect it to fit, as 24B-A4B Gemma model without CPU offloading got up to 18GB of VRAM usage
- zozbot234 6mo agoCPU-MoE still helps with mmap. Should not overly hurt token-gen speed on the Mac since the CPU has access to most (though not all) of the unified memory bandwidth, which is the bottleneck.