2 ms·
Get as much RAM as possible. 16GB is not enough. 32GB is enough to run quantized Mixtral, which is the current best openly licensed model. ... but who knows
by simonw 3y ago
Get as much RAM as possible.
16GB is not enough.
32GB is enough to run quantized Mixtral, which is the current best openly licensed model.
... but who knows what will emerge in the next 12 months?
I have 64GB and I'm regretting not shelling out for more.
Frustratingly you still have WAY more options for running interesting models on a Linux or Windows NVIDIA device, but I like Mac for a bunch of other reasons.
- eurekin 3y agoWhy not both? :) Bait aside, I'd love to read about how are you using those models. I'm mostly interested in code comprehension and meeting summarisation.
- simonw 3y agoI still mostly use GPT-4 to get actual work done, because until a Mixtral came along it felt like the local models were no competition. I'm going to bump up my usage of Mixtral a bit now to see how it feels for that kind of stuff.
- eurekin 3y agoAh, same here. Have been itching to make some use of those local llms for some time, always ending up in chatgpt as well. Although for those napkin like ideas gpt4 (including the turbo variant) get costly quickly.
- nextworddev 3y agoJust got 64gb as well but about to refund it for a bigger one after reading your comment :)
- Casteil 3y agoI run 7b/13b models pretty gracefully on a 16GB M1 Pro, but it does leave me wanting a little more headroom for other things like Docker and browser eating multiple gigabytes of ram themselves. Maybe keep an eye out for M1 / M2 deals with high ram config? I've seen 64GB MBPs lately for <$2300 (slickdeals.net)
- MilaM 3y agoThank you (and all the adjacent comments). I don't really need so much RAM for my regular work. Running the llms locally would only be for fun and experiments. I think 32GB might be the best middle ground for my needs and budget constraints. It's really a pity that you can't extend RAM in most Apple Silicon Macs and have to decide carefully upfront.
- amrrs 3y agoI ended up ordering a 36GB M3 for similar reason. I currently run Mistral and a few mistral derivatives using Ollama with decent inference speed on a 2019 Intel Mac 32GB. So I assumed the new one with 32ish should do a better job. I've tried vision model Llava as well, a bit more latency but works fine. With Apple's own Mlx things might improve .