3 ms·

I hope the other sizes are coming too(9B for me). Can't fit much context with this on a 36GB mac.

by abhikul0 6mo ago

pdyc 6mo ago
can you elaborate? you can use quantized version, would context still be an issue with it?
- nickthegreek 6mo ago
  context is always an issue with local models and consumer hardware.
  - pdyc 6mo ago
    correct but it should be some ratio of model size like if model size is x GB, max context would occupy x * some constant of RAM. For quantized version assuming its 18GB for Q4 it should be able to support 64-128k with this mac
    - abhikul0 6mo ago
      For the 9B model, I can use the full context with Q8_0 KV. This uses around ~16GB, while still leaving a comfortable headroom. Output after I exit the llama-server command: llama_memory_breakdown_print: | memory breakdown [MiB] | total free self model context compute unaccounted | llama_memory_breakdown_print: | - MTL0 (Apple M3 Pro) | 28753 = 14607 + (14145 = 6262 + 4553 + 3329) + 0 | llama_memory_breakdown_print: | - Host | 2779 = 666 + 0 + 2112 |
- abhikul0 6mo ago
  A usable quant, Q5_KM imo, takes up ~26GB[0], which leaves around ~6-7GB for context and running other programs which is not much. [0] https://huggingface.co/unsloth/Qwen3.5-35B-A3B-GGUF?show_file_info=Qwen3.5-35B-A3B-Q5_K_M.gguf https://huggingface.co/unsloth/Qwen3.5-35B-A3B-GGUF?show_fil...
mhitza 6mo ago
It's a MoE model and the A3B stands for 3 Billion active parameters, like the recent Gemma 4. You can try to offload the experts on CPU with llama.cpp (--cpu-moe) and that should give you quite the extra context space, at a lower token generation speed.
- dgb23 6mo ago
  Do I expect the same memory footprint from an N active parameters as from simply N total parameters?
  - deleted 6mo ago
    [deleted]
  - daemonologist 6mo ago
    No - this model has the weights memory footprint of a 35B model (you do save a little bit on the KV cache, which will be smaller than the total size suggests). The lower number of active parameters gives you faster inference, including lower memory bandwidth utilization, which makes it viable to offload the weights for the experts onto slower memory. On a Mac, with unified memory, this doesn't really help you. (Unless you want to offload to nonvolatile storage, but it would still be painfully slow.) All that said you could probably squeeze it onto a 36GB Mac. A lot of people run this size model on 24GB GPUs, at 4-5 bits per weight quantization and maybe with reduced context size.
- pdyc 6mo ago
  i dont get it, mac has unified memory how would offloading experts to cpu help?
  - bee_rider 6mo ago
    I bet the poster just didn’t remember that important detail about Macs, it is kind of unusual from a normal computer point of view. I wonder though, do Macs have swap, coupled unused experts be offloaded to swap?

I hope the other sizes are coming too(9B for me). Can't fit much context with this on a 36GB mac.

pdyc 6mo ago

can you elaborate? you can use quantized version, would context still be an issue with it?

nickthegreek 6mo ago
context is always an issue with local models and consumer hardware.
- pdyc 6mo ago
  correct but it should be some ratio of model size like if model size is x GB, max context would occupy x * some constant of RAM. For quantized version assuming its 18GB for Q4 it should be able to support 64-128k with this mac
  - abhikul0 6mo ago
    For the 9B model, I can use the full context with Q8_0 KV. This uses around ~16GB, while still leaving a comfortable headroom. Output after I exit the llama-server command: llama_memory_breakdown_print: | memory breakdown [MiB] | total free self model context compute unaccounted | llama_memory_breakdown_print: | - MTL0 (Apple M3 Pro) | 28753 = 14607 + (14145 = 6262 + 4553 + 3329) + 0 | llama_memory_breakdown_print: | - Host | 2779 = 666 + 0 + 2112 |
abhikul0 6mo ago
A usable quant, Q5_KM imo, takes up ~26GB[0], which leaves around ~6-7GB for context and running other programs which is not much. [0] https://huggingface.co/unsloth/Qwen3.5-35B-A3B-GGUF?show_file_info=Qwen3.5-35B-A3B-Q5_K_M.gguf https://huggingface.co/unsloth/Qwen3.5-35B-A3B-GGUF?show_fil...

mhitza 6mo ago

It's a MoE model and the A3B stands for 3 Billion active parameters, like the recent Gemma 4. You can try to offload the experts on CPU with llama.cpp (--cpu-moe) and that should give you quite the extra context space, at a lower token generation speed.

dgb23 6mo ago
Do I expect the same memory footprint from an N active parameters as from simply N total parameters?
- deleted 6mo ago
  [deleted]
- daemonologist 6mo ago
  No - this model has the weights memory footprint of a 35B model (you do save a little bit on the KV cache, which will be smaller than the total size suggests). The lower number of active parameters gives you faster inference, including lower memory bandwidth utilization, which makes it viable to offload the weights for the experts onto slower memory. On a Mac, with unified memory, this doesn't really help you. (Unless you want to offload to nonvolatile storage, but it would still be painfully slow.) All that said you could probably squeeze it onto a 36GB Mac. A lot of people run this size model on 24GB GPUs, at 4-5 bits per weight quantization and maybe with reduced context size.
pdyc 6mo ago
i dont get it, mac has unified memory how would offloading experts to cpu help?
- bee_rider 6mo ago
  I bet the poster just didn’t remember that important detail about Macs, it is kind of unusual from a normal computer point of view. I wonder though, do Macs have swap, coupled unused experts be offloaded to swap?