2 ms·
No idea. MLX is its own thing, optimized for Apple Silicon. Ollama uses GGUFs. https://ollama.com/library/gemma3:27b-it-qat https://ollama.com/library/gemma3:2
by simonw 2y ago
No idea. MLX is its own thing, optimized for Apple Silicon. Ollama uses GGUFs.
https://ollama.com/library/gemma3:27b-it-qat https://ollama.com/library/gemma3:27b-it-qat says it's Q4_0. https://huggingface.co/mlx-community/gemma-3-27b-it-qat-4bit https://huggingface.co/mlx-community/gemma-3-27b-it-qat-4bit says it's 4bit. I think those are the same quantization?
- jychang 2y agoThose are the same quant, but this is a good example of why you shouldn't use ollama. Either directly use llama.cpp, or use something like LM Studio if you want something with a GUI/easier user experience. The Gemma 3 17b QAT GGUF should be taking up ~15gb, not 22gb.