3 ms·
If you have a bit more memory, use the 6 bit quant, takes up about 26gb and has been shown to be very minimally lossy as opposed to 4bit. Also serve it as MLX
by brandall10 2y ago
If you have a bit more memory, use the 6 bit quant, takes up about 26gb and has been shown to be very minimally lossy as opposed to 4bit.
Also serve it as MLX from LMStudio, will speed things up 30% or so so your 6bit will have similar perf to the 4bit.
Getting about 12-13 tok/sec on my M3 Max 48gb.
- matwood 2y agoCan you link to the model you’re talking about? I can’t find the exact one using your description. Thanks!
- evrenesat 2y agohttps://huggingface.co/mlx-community/DeepSeek-R1-Distill-Qwen-32B-6bit https://huggingface.co/mlx-community/DeepSeek-R1-Distill-Qwe...
- thomasskis 2y agoEXO is also great for running the 6bit deepseek, plus it’s super handy to serve from all your devices simultaneously. If your dev team all has M3 Max 48gb machines, sharing the compute lets you all run bigger models and your tools can point at your local API endpoint to keep configs simple. Our enterprise internal IT has a low friction way to request a Mac Studio (192GB) for our team and it’s a wonderful central EXO endpoint. (Life saver when we’re generally GPU poor)