2 ms·

Anything good to run on Mac M5 Max with 48GB? is this even worth trying? so far I found the responses so slow compared to the paid subscriptions...

• seanmcdirmid 1mo ago
  I get up to 90 tokens / second with Jundot/Qwen3.6-35B-A3B-oQ6-mtp, on a M3 Max with 64GB. MoE so it is not a dense model, but that means it runs faster (also, mtp helps). It is a 30GB model, but you should be able to load it, otherwise try the 4-bit quant instead of the 6-bit quant, don't bother quanting your KV Cache (don't enable turboquant in oMLX), since that will slow you down. I'm not sure what that means on a M5 max, definitely faster, I don't know if it really plays into the strengths of the new chip design though.
  • coder-pm 1mo ago
    Thanks! I have to try that! Might be tight! Can it run in Claude Code? Are you loading it with Ollama?
    • seanmcdirmid 1mo ago
      As far as I know, if you want MTP you need to serve it with either MTPLX or oMLX. oMLX is more stable and user friendly. I’m using Goose but have tested Codex and OpenClaw among others. Codex worked fine but I couldn’t get web search to work with Qwen. OpenClaw I could get web search to work but its system prompt eats a lot of context tokens. Goose works for web search if you use SearchXNG, and has fairly slim system prompts. I’m still evaluating DeepSeek Harness and might move to that once it’s more stable. I tried other more obscure options but they had some flaws that made me settle on goose.
      • coder-pm 1mo ago
        Nice, I will give it a try! Thanks!
      • deleted 1mo ago
        [deleted]