3 ms·
MLX is slower than GGUFs on Macs. On my M1 Max macbook pro, the GGUF version bartowski/google_gemma-3-27b-it-qat-GGUF is 15.6gb and runs at 17tok/sec, whereas
by jychang 2y ago
MLX is slower than GGUFs on Macs.
On my M1 Max macbook pro, the GGUF version bartowski/google_gemma-3-27b-it-qat-GGUF is 15.6gb and runs at 17tok/sec, whereas mlx-community/gemma-3-27b-it-qat-4bit is 16.8gb and runs at 15tok/sec. Note that both of these are the new QAT 4bit quants.
- phaedrix 2y agoNo, in general mlx versions are always faster, ice tested most of them.
- 85392_school 2y agoWhat TPS difference are you getting?