3 ms·
A max with 64 GB of ram should be able to run this (I hope). I have to wait until an MLX model is available to really evaluate its speed, though.
by seanmcdirmid 2y ago
A max with 64 GB of ram should be able to run this (I hope). I have to wait until an MLX model is available to really evaluate its speed, though.
- mettamage 2y agoYep, it does that. I have 64 GB and was actually running 40 GB of other stuff.
- pickettd 2y agoLooks like MLX quants are released: - 4bit: https://huggingface.co/mlx-community/QwQ-32B-4bit https://huggingface.co/mlx-community/QwQ-32B-4bit - 6bit: https://huggingface.co/mlx-community/QwQ-32B-6bit https://huggingface.co/mlx-community/QwQ-32B-6bit
- seanmcdirmid 2y agoI downloaded the 8 bit quant last night but haven't had a chance to play with it yet.