2 ms·
Nice, I tried it too with oMLX — agreed, it seems very capable for coding! Was slightly underwhelmed by performance though. I got about ~24 t/s on the ternary
by ch_sm 3mo ago
Nice, I tried it too with oMLX — agreed, it seems very capable for coding! Was slightly underwhelmed by performance though. I got about ~24 t/s on the ternary version on my M2 Max 64. That’s quite a bit slower than Qwen A3B 35B (4 bit unsloth). How was perf for you?
- anon373839 3mo agoI had roughly the same performance on M3 Max 64: ~30 tokens/sec. Which is not terrible, but with the latest Lightning MTP optimization I am getting ~100 tokens/sec with Qwen 3.6 35B-A3B.
- ch_sm 3mo agoWHAT. that‘s amazing. thanks so much for sharing, i‘ve been looking for ways to speed a3b up for days. It‘s 11pm here but i‘ll try this right now
- anon373839 3mo agoI’m using the new oQe quants in case that matters.
- ch_sm 3mo agoi see! would you mind sharing your model string?
- anon373839 3mo ago[dead]
- midulan 2mo agoi'm getting similar toks on my 36gb m3 max. new omlx release (0.5.2) has dedicated decode kernels for the bonsai models — see if this improves perf for you.