3 ms·
I guess I won’t be needing that 512GB M3 Ultra after all.
by manmal 2y ago
I guess I won’t be needing that 512GB M3 Ultra after all.
- outside415 2y ago[dead]
- UncleOxidant 2y agoI think the Framework AI PC will run this quite nicely.
- Tepix 2y agoI think you want a lot of speed to make up for the fact that it's so chatty. Two 24GB GPUs (so you have room for context) will probably be great.
- rpastuszak 2y agoHow much vram do you need to run this model? Is 48 gb unified memory enough?
- daemonologist 2y agoThe quantized model fits in about 20 GB, so 32 would probably be sufficient unless you want to use the full context length (long inputs and/or lots of reasoning). 48 should be plenty.
- manmal 2y agoI‘ve tried the very early Q4 mlx release on an M1 Max 32GB (LM Studio @ default settings), and have run into severe issues. For the coding tasks I gave it, it froze before it was done with reasoning. I guess I should limit context size. I do love what I‘m seeing though, the output reads very similar to R1, and I mostly agree with its conclusions. The Q8 version has to be way better even.
- whitehexagon 2y agoDoes the Q8 fit within your 32GB (also using an M1 32GB)
- manmal 2y agoNo, Q4 just barely fits, and with a longer context sometimes things freeze. I definitely have to close Xcode.
- zamalek 2y ago39gb if you use a fp8 quantized model.[1] Remember that your OS might be using some of that itself. As far as I recall, Ollama/llama.cpp recently added a feature to page-in parameters - so you'll be able to go arbitrarily large soon enough (at a performance cost). Obviously more in RAM = more speed = more better. [1]: https://token-calculator.net/llm-memory-calculator https://token-calculator.net/llm-memory-calculator
- dulakian 2y agoI am using the Q6_K_L quant and it's running at about 40G of vram with the KV cache. Device 1 [NVIDIA GeForce RTX 4090] MEM[||||||||||||||||||20.170Gi/23.988Gi] Device 2 [NVIDIA GeForce RTX 4090] MEM[||||||||||||||||||19.945Gi/23.988Gi]
- brandall10 2y agoIt's enough for 6 bit quant with a somewhat restricted context length. Though based on the responses here, it needs sizable context to work, so we may be limited to 4 bit (I'm on an M3 Max w/ 48gb as well).
- seanmcdirmid 2y agoA max with 64 GB of ram should be able to run this (I hope). I have to wait until an MLX model is available to really evaluate its speed, though.
- mettamage 2y agoYep, it does that. I have 64 GB and was actually running 40 GB of other stuff.
- pickettd 2y agoLooks like MLX quants are released: - 4bit: https://huggingface.co/mlx-community/QwQ-32B-4bit https://huggingface.co/mlx-community/QwQ-32B-4bit - 6bit: https://huggingface.co/mlx-community/QwQ-32B-6bit https://huggingface.co/mlx-community/QwQ-32B-6bit
- seanmcdirmid 2y agoI downloaded the 8 bit quant last night but haven't had a chance to play with it yet.